Drives me nuts that this nomenclature caught on. This (judgement, not decision) is a bit better. I mean, sure they (Jev-style models) are useful for zero-shot classification, but they're only "judgement models" insofar as any (LLM, ML, mathematical optimization, heuristic, etc.) model that sets a downstream float / integer / boolean to commit resources (read: makes a decision) is a "judgement model".
If you actually want to make good, even optimal (provably) decisions, try reading some about mathematical optimization, or operations research more broadly. The above link is a good place to start.
"System One" is a mess of a name from the start, as is any classification that chooses arbitrary labels for non-arbitrary things (Type I vs II errors, etc.)
Drives me nuts that this nomenclature caught on. This (judgement, not decision) is a bit better. I mean, sure they (Jev-style models) are useful for zero-shot classification, but they're only "judgement models" insofar as any (LLM, ML, mathematical optimization, heuristic, etc.) model that sets a downstream float / integer / boolean to commit resources (read: makes a decision) is a "judgement model".
https://warrenpowell.org/decisionsdecisions/
If you actually want to make good, even optimal (provably) decisions, try reading some about mathematical optimization, or operations research more broadly. The above link is a good place to start.
"System One" is a mess of a name from the start, as is any classification that chooses arbitrary labels for non-arbitrary things (Type I vs II errors, etc.)
Really cool visual way to do demonstrate the point. Jev is so interesting to me, I just struggle to find good use cases in my work so far.
I agree with this.
Decisions/Judgements in LLMs are not logic gates like a deterministic piece of software, they are simulations cosplaying it more often than not.
Depending on how it's worded it's pretty remarkable how much it can improve, but it ultimately is not deterministic.