I noticed a few weeks ago it started being very bad at explaining things (even things itself was doing) and started committing absurd errors (like reading a test of 5 lines and not noticing there was an explicit mock created in one of those, then saying that the test was failing while it was not)
I fear this is just the classic "nerf the model just before we release a new version of it"
I know Opus is finished with the simple task I asked for when it starts outputting a 400-page lesson in verbosity (an opus?) complete with comparison tables, bullet-pointed lists of vaguely worded assertions, several self-blunder reports and a list of things it wants to mention but didn't touch yet but just say the word and it will.
It definitely has to count as output tokens when it blathers on endlessly like that, yes. I actually wrote a skill called /speak-normal and that helps but over a long context Opus will forget and start getting real wordy again over time. It can be frustrating.
For me, it works better than ever. When I once saw a quality decline, it was conflicting CLAUDE.md files for me (local vs. global) so basically my fault.
the usage and quality out of gpt 5.6 is on par, if not better in token usage. I would love to do a write-up on which tools are useful for what use-case, as AA keeps improving.
I use gemini for rewriting code docs, because the frontier models are so verbose when it comes to writing text
I mean, only yesterday Fable at max effort forgot to commit and push half the changes in two files and didn't mention it until I found out with git status.
It made half the changes, committed and pushed, then it made the other half of the changes on the same two files as before and... just stopped and reported back with a cheerful "all good, all done and pushed".
LLMs are nondeterministic, good luck trying to measure performance at all, much less over time.
Lets say it has gotten worse? What are you going to do about it? Jump to Codex? Then what happen if you perceive that to be getting worse? Jump back to Claude? One of the many problems with these tools.
I noticed a few weeks ago it started being very bad at explaining things (even things itself was doing) and started committing absurd errors (like reading a test of 5 lines and not noticing there was an explicit mock created in one of those, then saying that the test was failing while it was not)
I fear this is just the classic "nerf the model just before we release a new version of it"
I know Opus is finished with the simple task I asked for when it starts outputting a 400-page lesson in verbosity (an opus?) complete with comparison tables, bullet-pointed lists of vaguely worded assertions, several self-blunder reports and a list of things it wants to mention but didn't touch yet but just say the word and it will.
When it's overly verbose like that. Does that count against your usage as well? Also during its thinking does it output hidden thinking tokens?
I am now wondering
It definitely has to count as output tokens when it blathers on endlessly like that, yes. I actually wrote a skill called /speak-normal and that helps but over a long context Opus will forget and start getting real wordy again over time. It can be frustrating.
Claude has become worse; it is condescending, robotic, and responses are peppered with needless words.
I canceled my subscription and moved back to ChatGPT and happier so far.
For me, it works better than ever. When I once saw a quality decline, it was conflicting CLAUDE.md files for me (local vs. global) so basically my fault.
the usage and quality out of gpt 5.6 is on par, if not better in token usage. I would love to do a write-up on which tools are useful for what use-case, as AA keeps improving.
I use gemini for rewriting code docs, because the frontier models are so verbose when it comes to writing text
I mean, only yesterday Fable at max effort forgot to commit and push half the changes in two files and didn't mention it until I found out with git status.
It made half the changes, committed and pushed, then it made the other half of the changes on the same two files as before and... just stopped and reported back with a cheerful "all good, all done and pushed".
it becomes really slow for me recent days, a simple task would take lots of time to work on. A simple commit request takes forever
LLMs are nondeterministic, good luck trying to measure performance at all, much less over time.
Lets say it has gotten worse? What are you going to do about it? Jump to Codex? Then what happen if you perceive that to be getting worse? Jump back to Claude? One of the many problems with these tools.
Gemini is good at Germany?
It speaks Germany perfectly. And even English sometimes.
It's good at polishing English text!
Claude Opus