> But you don’t care about code quality anymore right.
Why did we ever? Because it made it more feasible to maintain and reduced bugs. Well structured, designed code bases are simply far easier to work with.
I'm sure we've all had greenfield AI projects where we just let it go nuts doing whatever it wants, then all of a sudden you need what should be a stupid easy change, and it just falls over completely or becomes borderline impossible. Each change needs 5+ other changes until you basically can't touch the thing. The difference is, AI will keep trying until you run out of money or you give up, but this way of programming is bad and we labeled it bad for a reason, and I don't know why as an industry we're thinking it's no longer important.
I mean, code complexity is still something that should be accounted for and frowned upon. I think the issue is just that the tools we have to measure complexity (or linting, or coverage, or flakiness, etc) currently aren't integrated into the CI/CD in any meaningful way other than 'here's a comment on your PR.'
I think about test-driven development and how good versions of that were socialized norms, processes and checks that happened prior to outputting code. Just telling an LLM to do TDD just creates a hellish CI latency tail with too many tests - you have to have taste and discernment there and none of that's built-in. Context rot is still too much of an issue especially if you're on a distributed MS model. I've had a little luck with whole system ADRs helping (with a lot of skills + active prompting) but the context still degrades over longer sessions or the further into a sprint you get.
I've seen some preliminary 'innovations' like this with LLM-driven shared wikis but it's all very hand-wavey and not standardized across the industry. It really feels like people are just waiting for these frontier labs to fully solve code quality (my gut says that the scale of improvement is logarithmic, not exponential, at least not until there's actual revolutionary research on LLM outputs at the scale of something like tensors) or they hope that some outfit with a fully vibe-coded website that promises to solve all of these things will solve all of these things.
In its current state, LLMs produce code that is on par with the average developer. This is not necessarily bad since most firms can only afford to hire average developers, the difference in quality is negligible. And on the flip side, by producing the code it does, AI also raises the floor by eliminating code smells typical of below-average developers, so there's that.
Over 12-18 months of moving my teams to LLM as the IDE I've observed they DO NOT equal a mid-level engineer on their own. More importantly, over time they code reduces to more that of a junior self-taught engineer who has no concept of software engineering (delivery as a team vs. raw academic).
I've built around Claude (althought the infra is LLM agnostic) and the code it produces is okay. You can invest and get better. However, that was the same BEFORE llm generated code.
I'm finding that my investment motions, governance motions are more about medium change, not much else.
Standardards, guidelines, et al are now stored in .md vs. wiki, docs, etc.
Code reviews are still code reviews.
Release planning is still releaes planning.
All the same motions. JUST LOTS MORE OF THEM.
But we've scaled quality throughput before. It's just with a new IDE.
More like a genius junior developer who's been home-schooled by the web for better and worse. Hard-working, often impressive, but missing some common sense, occasionally spouts nonsense, and needs guidance to make sure they don't go off the rails.
"We need formalized processes, and proven, repeatable control mechanisms to enforce the quality of whatever is coming out of these tools. These do not exist yet. They might emerge one day, but they have not yet3. Until then, I won’t just lay into it blindly. If you consider yourself a software engineer, you should probably give it a minute too."
The same argument cuts both ways. I have seen some horrid code (that I have had to fix) made by humans.
Yeah, but that bad code also had the chance to be caught and fixed in review. If you get rid of reviewing code (human-written, AI-written, squirrel-written, whatever) you get rid of that chance. Hence the not trusting it blindly part.
We've been scaling quality alongside throuhgput for decades (at least since I joined the biz in 1999). AI is a large step forward in the automation game, but it's not new, nor are it's challenges.
Are other orgs not relying on the fundamentals of software engineering and adapting to this form of generated code?
PS: I don't allow one-shots. Same decomposition and quality assurance motions. The timeline has just shrunk, not the steps, nor the focus. The medium for documenting and governamce have shifted, same strategies and problems.
In my experience, the clankers can write great code when you give it a hole with the right shape in the bigger picture of code that looks how you like. I had the experience of setting a clanker loose on Wizard to go implement a Wasm proposal[1] that had reached stage 4, and it wrote the canonical V3 interpreter implementation, fast assembly interpreter implementation, and JIT support, as well as code validation and fitting the rest into the system, in just 40 minutes, complete with tests. The VM code was canonical. There wasn't a line I would have written otherwise. In the end it was only 144 lines. But it was the right 144 lines.
Clankers can do things right when they are given clear instructions and good context. They go bad when they are given agency and no oversight. What I have seen repeatedly is that even small amounts of correction to turn a refactoring this way or that way or make a better modularity decision, or to enforce a boundary that wasn't written down, helps a lot. Clankers don't know about your implicit information hiding going on. They don't know you've been working for years to get rid of a bad code smell that spread faster than you could stamp it out, or that you've been working on the same codebase for 10+ years and only now understand how you'd like to structure it.
Some people use the clanker to get something out of nothing. As engineers, we should be using them to engineer the heck out of things that we didn't have human effort available for. Things like refactoring, testing, cleaning up documentation, doing optimizations, simplifying, introducing proper interfaces, removing improper interfaces, etc.
[1] The wide-arithmetic proposal, which added 4 instructions with very well-specified behavior.
> Ok but that sounds like a user issue: you’re bad at prompting and you’re putting it on the LLM.
>Shit, I forgot to add you're a distinguished principal 10x engineermaxxer who's unfailable, make no mistakes and double-check your shit.
I know he's being funny and explains later but obviously that's not enough. There's lots of guardrails you can put on code delivery pipelines that help. In my experience the code produces good results way higher than 95%. 95% would be horrific.
> And that’s what really drives me nuts: when that particularly stubborn junior dev does the same mistake for the 3rd time, you get coachy for a minute, make sure they learn, and that actually helps them level up. Clankers don’t learn. They can’t, because the underlying technology cannot learn.
This is why I left IT and went for an EE degree program. I actually enjoy helping people to grow, I always did. (I've had a colorful career involving a lot of different industries, so chances are high I might have picked up a few things here and there)
But when there are no juniors any more, only clankers, what the fuck is even my purpose other than making money for some American slave drivers uhhh shareholders.
I'll miss the era when we could care about the craft of coding. I'll probably continue to do so in my spare time. Hobbyist woodworkers and hand-crafted furniture makers still exist, even if 99% of all chairs are made by machine now.
But code craft not the job. The job is to produce business value. Code craft used to aid that, but now it's an impediment.
Your job is no longer to produce code - it's to build systems that produce code. Tests and CI and Linters and custom lint rules and adversarial review and agent docs and automations and the next dozen tools we come up with as part of figuring out best practices around all of this. If you're getting woken up by agent-written code, the fix isn't to review every line of code - the fix is to improve your system so that class of error doesn't get written.
It's understandable if you don't want to do that work. It's almost entirely different from the work of reading and writing code. But if you don't, you're delivering less business value than someone who will. They don't employ hand woodworkers at Ikea.
> But code craft not the job. The job is to produce business value. Code craft used to aid that, but now it's an impediment.
That is false. Code craft is still very much an aid to business value. If you just turn an LLM loose and don't review its work, you get a buggy mess over time. If you use an LLM, you must review and understand each line of code or you are sabotaging the quality of your work.
My whole thesis is we've been doing this with humans since the beginning. I'm not finding anything unique about LLM generated code. You gotta define standards the "developer" can read and implement. You gotta review the code for compliance. You gotta test the code for quality. You gotta perfomrance test for scalability. Who gives a flying turkey whether a meat-puppet (human) or a clanker (llm) wrote the code? It's still just code.
> But you don’t care about code quality anymore right.
Why did we ever? Because it made it more feasible to maintain and reduced bugs. Well structured, designed code bases are simply far easier to work with.
I'm sure we've all had greenfield AI projects where we just let it go nuts doing whatever it wants, then all of a sudden you need what should be a stupid easy change, and it just falls over completely or becomes borderline impossible. Each change needs 5+ other changes until you basically can't touch the thing. The difference is, AI will keep trying until you run out of money or you give up, but this way of programming is bad and we labeled it bad for a reason, and I don't know why as an industry we're thinking it's no longer important.
I mean, code complexity is still something that should be accounted for and frowned upon. I think the issue is just that the tools we have to measure complexity (or linting, or coverage, or flakiness, etc) currently aren't integrated into the CI/CD in any meaningful way other than 'here's a comment on your PR.'
I think about test-driven development and how good versions of that were socialized norms, processes and checks that happened prior to outputting code. Just telling an LLM to do TDD just creates a hellish CI latency tail with too many tests - you have to have taste and discernment there and none of that's built-in. Context rot is still too much of an issue especially if you're on a distributed MS model. I've had a little luck with whole system ADRs helping (with a lot of skills + active prompting) but the context still degrades over longer sessions or the further into a sprint you get.
I've seen some preliminary 'innovations' like this with LLM-driven shared wikis but it's all very hand-wavey and not standardized across the industry. It really feels like people are just waiting for these frontier labs to fully solve code quality (my gut says that the scale of improvement is logarithmic, not exponential, at least not until there's actual revolutionary research on LLM outputs at the scale of something like tensors) or they hope that some outfit with a fully vibe-coded website that promises to solve all of these things will solve all of these things.
In its current state, LLMs produce code that is on par with the average developer. This is not necessarily bad since most firms can only afford to hire average developers, the difference in quality is negligible. And on the flip side, by producing the code it does, AI also raises the floor by eliminating code smells typical of below-average developers, so there's that.
Over 12-18 months of moving my teams to LLM as the IDE I've observed they DO NOT equal a mid-level engineer on their own. More importantly, over time they code reduces to more that of a junior self-taught engineer who has no concept of software engineering (delivery as a team vs. raw academic).
I've built around Claude (althought the infra is LLM agnostic) and the code it produces is okay. You can invest and get better. However, that was the same BEFORE llm generated code.
I'm finding that my investment motions, governance motions are more about medium change, not much else.
Standardards, guidelines, et al are now stored in .md vs. wiki, docs, etc.
Code reviews are still code reviews.
Release planning is still releaes planning.
All the same motions. JUST LOTS MORE OF THEM.
But we've scaled quality throughput before. It's just with a new IDE.
> on par with the average developer
More like a genius junior developer who's been home-schooled by the web for better and worse. Hard-working, often impressive, but missing some common sense, occasionally spouts nonsense, and needs guidance to make sure they don't go off the rails.
"We need formalized processes, and proven, repeatable control mechanisms to enforce the quality of whatever is coming out of these tools. These do not exist yet. They might emerge one day, but they have not yet3. Until then, I won’t just lay into it blindly. If you consider yourself a software engineer, you should probably give it a minute too."
The same argument cuts both ways. I have seen some horrid code (that I have had to fix) made by humans.
Yeah, but that bad code also had the chance to be caught and fixed in review. If you get rid of reviewing code (human-written, AI-written, squirrel-written, whatever) you get rid of that chance. Hence the not trusting it blindly part.
We've been scaling quality alongside throuhgput for decades (at least since I joined the biz in 1999). AI is a large step forward in the automation game, but it's not new, nor are it's challenges.
Are other orgs not relying on the fundamentals of software engineering and adapting to this form of generated code?
PS: I don't allow one-shots. Same decomposition and quality assurance motions. The timeline has just shrunk, not the steps, nor the focus. The medium for documenting and governamce have shifted, same strategies and problems.
Am I really on an island with this?
In my experience, the clankers can write great code when you give it a hole with the right shape in the bigger picture of code that looks how you like. I had the experience of setting a clanker loose on Wizard to go implement a Wasm proposal[1] that had reached stage 4, and it wrote the canonical V3 interpreter implementation, fast assembly interpreter implementation, and JIT support, as well as code validation and fitting the rest into the system, in just 40 minutes, complete with tests. The VM code was canonical. There wasn't a line I would have written otherwise. In the end it was only 144 lines. But it was the right 144 lines.
Clankers can do things right when they are given clear instructions and good context. They go bad when they are given agency and no oversight. What I have seen repeatedly is that even small amounts of correction to turn a refactoring this way or that way or make a better modularity decision, or to enforce a boundary that wasn't written down, helps a lot. Clankers don't know about your implicit information hiding going on. They don't know you've been working for years to get rid of a bad code smell that spread faster than you could stamp it out, or that you've been working on the same codebase for 10+ years and only now understand how you'd like to structure it.
Some people use the clanker to get something out of nothing. As engineers, we should be using them to engineer the heck out of things that we didn't have human effort available for. Things like refactoring, testing, cleaning up documentation, doing optimizations, simplifying, introducing proper interfaces, removing improper interfaces, etc.
[1] The wide-arithmetic proposal, which added 4 instructions with very well-specified behavior.
> Ok but that sounds like a user issue: you’re bad at prompting and you’re putting it on the LLM.
>Shit, I forgot to add you're a distinguished principal 10x engineermaxxer who's unfailable, make no mistakes and double-check your shit.
I know he's being funny and explains later but obviously that's not enough. There's lots of guardrails you can put on code delivery pipelines that help. In my experience the code produces good results way higher than 95%. 95% would be horrific.
We provide a service on a free market. So, that’s just great, as long as you find someone to pay you to read your clankers’ code.
> And that’s what really drives me nuts: when that particularly stubborn junior dev does the same mistake for the 3rd time, you get coachy for a minute, make sure they learn, and that actually helps them level up. Clankers don’t learn. They can’t, because the underlying technology cannot learn.
This is why I left IT and went for an EE degree program. I actually enjoy helping people to grow, I always did. (I've had a colorful career involving a lot of different industries, so chances are high I might have picked up a few things here and there)
But when there are no juniors any more, only clankers, what the fuck is even my purpose other than making money for some American slave drivers uhhh shareholders.
[dead]
I'll miss the era when we could care about the craft of coding. I'll probably continue to do so in my spare time. Hobbyist woodworkers and hand-crafted furniture makers still exist, even if 99% of all chairs are made by machine now.
But code craft not the job. The job is to produce business value. Code craft used to aid that, but now it's an impediment.
Your job is no longer to produce code - it's to build systems that produce code. Tests and CI and Linters and custom lint rules and adversarial review and agent docs and automations and the next dozen tools we come up with as part of figuring out best practices around all of this. If you're getting woken up by agent-written code, the fix isn't to review every line of code - the fix is to improve your system so that class of error doesn't get written.
It's understandable if you don't want to do that work. It's almost entirely different from the work of reading and writing code. But if you don't, you're delivering less business value than someone who will. They don't employ hand woodworkers at Ikea.
I’m sure out of the many attempts to automate the production of software and get rid of those pesky, expensive devs, THIS one will work.
Tell me again how the code quality of the Linux kernel doesn't matter, only its business value.
> But code craft not the job. The job is to produce business value. Code craft used to aid that, but now it's an impediment.
That is false. Code craft is still very much an aid to business value. If you just turn an LLM loose and don't review its work, you get a buggy mess over time. If you use an LLM, you must review and understand each line of code or you are sabotaging the quality of your work.
My whole thesis is we've been doing this with humans since the beginning. I'm not finding anything unique about LLM generated code. You gotta define standards the "developer" can read and implement. You gotta review the code for compliance. You gotta test the code for quality. You gotta perfomrance test for scalability. Who gives a flying turkey whether a meat-puppet (human) or a clanker (llm) wrote the code? It's still just code.