Of course quality isn't affected... that much (once AGENTS.md is maintained). Where I imagine the difference would been seen the most is in the number of tool calls, and whether the results are useful or not. My prediction is that a codebase with comments - which are kept updated - leads to fewer useless tool calls over time. The benefit stacks, particularly as codebase complexity increases, and across sessions. I've seen enough evidence of this in my projects that I'd rather let the agent comment as it likes (though I imagine it could be even better if there was guidance on ensuring comments don't go stale, as an agent in a fresh session hitting a stale comment can be worse than no comment).
> Almost a third of our source code was comments. I deleted all of them and asked an AI agent to build the same features on both versions of the codebase, three times each. The code it wrote was just as good with or without comments.
Seems like this might be another example of The Bitter Lesson [1]:
"The bitter lesson is the observation in artificial intelligence that, in the long run, general approaches that scale with available computational power tend to outperform ones based on domain-specific understanding because they are better at taking advantage of the falling cost of computation over time."
AIs don't need comments that are designed to work around human cognitive limitations. They're quite happy to just crunch the code and extract the meaning.
Nitpicking, but I don't think it's correct to stretch the term to this. As the definition you quoted mentions, the bitter lesson is about approaches to AI development, not about details of applied LLM usage.
> They're quite happy to just crunch the code and extract the meaning.
And burn X more tokens on rediscovering relevant surrounding context, rather than having it potentially readily available - or at least pointers to it - in the comments.
Makes sense if the comments were AI generated as the author writes since AI during reading the code has roughly the same context as with which it produces the comments. I would be more curious for a test with human written comments.
This is really interesting, my gut instinct was that comments would always be helpful to AI clarifying the broader picture of what code is doing, and defining a spec to police what code should be doing.
I think there are still contexts where comments would be superior like when they feed into an OpenAPI spec where the AI or person can't see the code, and information that bubbles up through Intellisense (for humans anyway).
LLMs usually write comments which the code afterwards exactly reflects - it might help them at the time, but these "waypoint comments" as I call them are almost always useless afterwards.
Anthropics models have been steered quite heavily to emit long and pointlessly verbose comments, and my best guess as to why is to burn output tokens. The reason I've stopped using them is even when explicitly instructed to "NEVER LEAVE ANY COMMENTS WHATSOEVER" or a similar command, either in prompt form or as a CLAUDE.md/memory, they _still do_. They sometimes print a longer comment box than the function itself. At one point I even caught Fable 5 emitting comments and then _removing them_ manually instead of never emitting them in the first place.
// `const` now: the entry is no longer reassigned here, because
// `ensureGameRender()` moved inside the render callback below and
// its upgraded copy is consumed there.
* Phase 5 (boot decomposition) will move the runtime facade
* assembly here as well — a single `attachPublicApi(services)`
* function that wires every `wp.os.*` method in one place.
* Until then, the runtime assembly continues to live in
* `src/desktop.ts`.
* Phase 3 only ships drag-within-folder. Cross-folder drags
* (drop into a folder, drop out to root) and the Recycle Bin
* drop integration land in later phases.
// v1: no-op. Phase 4 will populate this from the push payload.
// We claim the event so a hosting environment that signs us up
// for a push subscription before the v2 PR lands doesn't see
// silently-dropped pushes — the empty handler still satisfies
// `Notification` capability checks on browsers that gate them.
There is a certain taste to not drop comments like this, but there is a tendency to do this in AI. They drop references to phases of work and outdated architecture in ways that aren't helpful and then don't get cleaned up.
This seems believable! And the n=3 of it all seems very wise (though n=3 alone isn't quite enough to speak confidently of "noise" AFAIU).
Two important caveats, though, before we rush to universalize this:
1. It's based on the smartest model ever released to the public in the history of humanity, Opus 5.5. So if this is true for your codebase, it probably only became so ~post-Mythos.
2. It seems like the hard task isn't really hard enough -- the "trick" is something about PHP, and sounds common enough that it would be well baked into each model. Regardless, there are far, far easier to miss gotchas than "fails on reload", especially in a large codebase where a post-mortem are written about correct code that just runs slower all of a sudden or something, much less cases where you have to rely on users (or worse, metrics) to uncover such issues.
When I joined Google, they gave me the baffling instruction to try to write as few comments as possible. I've since learned that's a semi-popular approach, based on the logic that "the code should speak for itself". I'm just a baby engineer (4 more months to make it onto 30-under-30!), but I find that to be more about self-esteem for seniors and feeling like a craftsman than it is about real, measured tradeoffs.
Plus, comments can A) be documentation[1] and B) be beautiful[2] ;)
This is so interesting! I would like to see the same experiment, but with these variations:
- All the comments removed (the blog post)
- Half the comments removed
- 10% of the comments replaced with slightly incorrect comments
- 10% of the comments replaced with very incorrect comments
- 10% of the comments replaced with slightly incorrect comments
- 90% of the comments removed, 10% remain but are very incorrect
And variations like that
Test to see where the inflection point is for correctness, token usage and other things
Of course quality isn't affected... that much (once AGENTS.md is maintained). Where I imagine the difference would been seen the most is in the number of tool calls, and whether the results are useful or not. My prediction is that a codebase with comments - which are kept updated - leads to fewer useless tool calls over time. The benefit stacks, particularly as codebase complexity increases, and across sessions. I've seen enough evidence of this in my projects that I'd rather let the agent comment as it likes (though I imagine it could be even better if there was guidance on ensuring comments don't go stale, as an agent in a fresh session hitting a stale comment can be worse than no comment).
> Almost a third of our source code was comments. I deleted all of them and asked an AI agent to build the same features on both versions of the codebase, three times each. The code it wrote was just as good with or without comments.
Seems like this might be another example of The Bitter Lesson [1]:
"The bitter lesson is the observation in artificial intelligence that, in the long run, general approaches that scale with available computational power tend to outperform ones based on domain-specific understanding because they are better at taking advantage of the falling cost of computation over time."
AIs don't need comments that are designed to work around human cognitive limitations. They're quite happy to just crunch the code and extract the meaning.
[1] https://en.wikipedia.org/wiki/Bitter_lesson
Nitpicking, but I don't think it's correct to stretch the term to this. As the definition you quoted mentions, the bitter lesson is about approaches to AI development, not about details of applied LLM usage.
On reflection I think you're correct
> They're quite happy to just crunch the code and extract the meaning.
And burn X more tokens on rediscovering relevant surrounding context, rather than having it potentially readily available - or at least pointers to it - in the comments.
Why does my browser lag scrolling a text article... awful.
Makes sense if the comments were AI generated as the author writes since AI during reading the code has roughly the same context as with which it produces the comments. I would be more curious for a test with human written comments.
This is really interesting, my gut instinct was that comments would always be helpful to AI clarifying the broader picture of what code is doing, and defining a spec to police what code should be doing.
I think there are still contexts where comments would be superior like when they feed into an OpenAPI spec where the AI or person can't see the code, and information that bubbles up through Intellisense (for humans anyway).
LLMs usually write comments which the code afterwards exactly reflects - it might help them at the time, but these "waypoint comments" as I call them are almost always useless afterwards.
Anthropics models have been steered quite heavily to emit long and pointlessly verbose comments, and my best guess as to why is to burn output tokens. The reason I've stopped using them is even when explicitly instructed to "NEVER LEAVE ANY COMMENTS WHATSOEVER" or a similar command, either in prompt form or as a CLAUDE.md/memory, they _still do_. They sometimes print a longer comment box than the function itself. At one point I even caught Fable 5 emitting comments and then _removing them_ manually instead of never emitting them in the first place.
Were the comments accessible in the bit history? If so, I wonder if the agent accessed it.
Comments and documents written by agents of are poor quality so its not surprising that deleting them is fine.
They tend to be wordy, long, and leak chatbot history or project management in unimportant ways that just distract.
I could point this out in my own code, but I don't need to since a vibe coded GitHub repo is linked in this very article.
> https://github.com/WordPress/openstation/blob/dd21eb131a2029...
> https://github.com/WordPress/openstation/blob/dd21eb131a2029... > https://github.com/WordPress/openstation/blob/dd21eb131a2029... > https://github.com/WordPress/openstation/blob/dd21eb131a2029... There is a certain taste to not drop comments like this, but there is a tendency to do this in AI. They drop references to phases of work and outdated architecture in ways that aren't helpful and then don't get cleaned up.This seems believable! And the n=3 of it all seems very wise (though n=3 alone isn't quite enough to speak confidently of "noise" AFAIU).
Two important caveats, though, before we rush to universalize this:
1. It's based on the smartest model ever released to the public in the history of humanity, Opus 5.5. So if this is true for your codebase, it probably only became so ~post-Mythos.
2. It seems like the hard task isn't really hard enough -- the "trick" is something about PHP, and sounds common enough that it would be well baked into each model. Regardless, there are far, far easier to miss gotchas than "fails on reload", especially in a large codebase where a post-mortem are written about correct code that just runs slower all of a sudden or something, much less cases where you have to rely on users (or worse, metrics) to uncover such issues.
When I joined Google, they gave me the baffling instruction to try to write as few comments as possible. I've since learned that's a semi-popular approach, based on the logic that "the code should speak for itself". I'm just a baby engineer (4 more months to make it onto 30-under-30!), but I find that to be more about self-esteem for seniors and feeling like a craftsman than it is about real, measured tradeoffs.
Plus, comments can A) be documentation[1] and B) be beautiful[2] ;)
[1]: https://github.com/doering-ai/my-basis/blob/main/docs/regex....
[2]: https://doering.ai/work/sublime-python-syntax/