Measures how models communicate rather than whether they are correct: 30 judged personality traits rolled up into 8 indexes, plus 21 measured communication characteristics.
I found the rating for Gemini models to track my experience. in addition to their extreme sycophancy, the information they provided (at least in the online chat) were more often than not, counterproductive. They act extremely confidently, and yield to any kind of pushback, requiring great care in prompting them to be able to get them to output useful alternatives.
This made the Gemini model family suboptimal for the purpose of investigating solutions to complex issues.
And their sycophancy rating tracks my experience, as I would groan loudly every time I read Gemini's answers and how it tries to nonsensically flatter me for asking simple, obvious questions.
At the other end, Deepseek is really good at pointing out contradictions, issues with my reasoning. It's as good as Opus (and vastly cheaper)
Where Deepseek loses out (in this study) compared to Anthropic and Open AI is "Slop", i.e. " How much generic filler or AI-flavored padding could be removed without losing substance?" though in my experience it didn't seem THAT different from Opus
The Organization category is worth a deeper look for anyone interested, because the results in this category vary wildly making the "overall" rating not very useful. e.g. GPT models are very structured i.e. " ideas organized in a logical sequence with clear progression" BUT not scannable "Can a reader skim the response and quickly find what they need? "
and while Deepseek scores highly in the overall rating for organization, its individual scores are pretty mediocre especially for Structure and Scannability, winning on the "bottom line summary" category.
PS: For fun, add "GPT-4o (Nov)" to the list. It's hilarious.
Measures how models communicate rather than whether they are correct: 30 judged personality traits rolled up into 8 indexes, plus 21 measured communication characteristics.
So my thoughts about the results here:
I found the rating for Gemini models to track my experience. in addition to their extreme sycophancy, the information they provided (at least in the online chat) were more often than not, counterproductive. They act extremely confidently, and yield to any kind of pushback, requiring great care in prompting them to be able to get them to output useful alternatives.
This made the Gemini model family suboptimal for the purpose of investigating solutions to complex issues.
And their sycophancy rating tracks my experience, as I would groan loudly every time I read Gemini's answers and how it tries to nonsensically flatter me for asking simple, obvious questions.
At the other end, Deepseek is really good at pointing out contradictions, issues with my reasoning. It's as good as Opus (and vastly cheaper)
Where Deepseek loses out (in this study) compared to Anthropic and Open AI is "Slop", i.e. " How much generic filler or AI-flavored padding could be removed without losing substance?" though in my experience it didn't seem THAT different from Opus
The Organization category is worth a deeper look for anyone interested, because the results in this category vary wildly making the "overall" rating not very useful. e.g. GPT models are very structured i.e. " ideas organized in a logical sequence with clear progression" BUT not scannable "Can a reader skim the response and quickly find what they need? "
and while Deepseek scores highly in the overall rating for organization, its individual scores are pretty mediocre especially for Structure and Scannability, winning on the "bottom line summary" category.
PS: For fun, add "GPT-4o (Nov)" to the list. It's hilarious.