I like the analogy. It feels right. But I don't think it actually is, because it leans hard on the _feeling_ of failure, vs. the actual failure.
The failure that actually matters here is product failure. Angry users. Security incidents. Downtime. Broken workflows.
Stressed and uncomfortable developers have, historically, been a leading indicator of product failure. Software is complex and when devs can't handle the breadth of ownership + the depth of expertise at the same time, it means bad work that leads to product failure. It's not clear that agents fail in the same way, or that our collective intuition about how and when to trust agents is sufficiently developed to predict product failure in the same way.
As anecdata: I know a bunch of growth-SaaS/last-batch-of-IPO companies where the devs are all stressed, the feature ship rate is up, and incidents are down. All at the same time.
So yes, quick learners and just-in-time depth is important, but so is intuition about where discomfort is going to produce true failure. The discomfort isn't a feeling to throw away, simply trusting that you'll learn everything in time. Ownership requires holding a model of the system in your head and proactively steering where you dive.
I think that this was true pre-attention economy. If you look at security incidents and other prod issues, they rarely move the needle today. I work at a hyperscalar and we have been shipping 40% more bugs to prod since 2023 year-to-year, yet user growth and revenue is up about 25% similarly due to more features and more ads.
I think that the whole premise of that failure affects things is based on a older model. You kind of assume that:
1) users are able to easily gain information about alternatives. We have seen that since LLMs replaced search engines, we have much less churn.
2) users are not behaviorally addicted/pushed into patterns. We have patterns that utilize users to keep using the app by exploiting their cognitive weaknesses
3) users have the mental fortitude and stamina to learn new tools instead of accepting flaws in the one they are used to
Some failures and bugs in prod don't matters as much
Almost the entire structure of software oriented companies, and why devs are expensive, are to prevent these sorts of errors. That job has gotten harder, and in some places impossible as management forces more trust on LLMs and reduces the capacity of devs to look. This is less of a problem in existing products in production and existing cultures - revenue is protected. Eventually those structures break down, as does the user perception of quality. Once the brands have taken enough knocks, it accelerates.
I agree that not all failures matter (most don't, in fact). On the other end of the spectrum, I know of a bunch of seed/series A startups that basically operate under a model where they ship everything and fix the stuff people complain about. As long as you do that fast enough, the median user experience is pretty good, because only the first n% actually see the problem before a fix is live.
I'll go out on a limb and guess that at a hyperscaler, shipping a bug to prod means hitting canary or 1% of traffic. Seems fine!
I think my point still stands, even as the shape of failures shift. There's certain ones that still hurt (data corruption, exploited security holes, etc.) because they're hard to notice. It might even be that the primary goal of ownership is to develop intuition about where those failures are likely to happen.
> Ownership requires holding a model of the system in your head and proactively steering where you dive.
Absolutely, and the feeling of discomfort is a signal that you need to dive, a signal that your mental models might need updating. I think that builds up over time through many dives
Site is down, but one thing in particular that needs close attention: db schema. I've found that LLMs make awful normalization decisions; you can limit the fallout somewhat by modularizing database access, bit it still inevitably bleeds over into the object model etc. On AI heavy projects I tend to pause feature development every few days for code & schema review, and subsequent cleanups.
Just do the schema by hand. It forces so much downstream alignment that you can almost prefer to use the schema and any associated comments over prompting the model directly.
If you don't know what your schema should be, then I think this is the dead canary in the vibe coding coal mine. Back up to the design phase and talk to the robot about the domain until you have a really strong conviction about its tuples and relations.
This matches my own experience, pre-AI. As I've moved from IC to having a team over the years, more code was being written than I could possibly understand at the depth I would have if I'd written it myself.
The skill I learned is exactly as you describe: understand the system. For me, this means "boxes and lines", the components and how they interact. I've encouraged developers on my team to make sequence diagrams part of their deliverable, because it helps both of us put the code in the context of the system.
The boxes can be opaque because I know the boundaries and expected inputs and outputs. This tells me when I need to go in a box: is it sending a malformed message? I can trace that back into the code.
I think that's the skill we have to cultivate now that coding agents produce far more than we can review line-by-line. The system view is the map, and the boundaries tell us where to go hunting inside the box.
I've been trying to adopt this approach and struggling. Any advice on how to learn this sort of systems approach? It doesn't seem to come very naturally to me
> Most of the successful engineers and engineering leaders I have had the pleasure of working with were able to hold a discussion on almost any topic, whether technical or not
Haha yes I've seen that too, I think the distinction is that it is not "fluent on everything" but rather "curious about everything". The best developers I know will happily admit when they don't know something and look it up, no matter the topic.
Mostly agreed with just in time learning. This is not just something that can be learned from the code though, this may require a huge amount of code context across multiple repositories and configuration files, as well as a knowledge of history, process, other tribal knowledge etc. So to me that's still the really difficult part of the job and AI definitely helps but doesn't entirely solve this. I see it often wants to solve some issue at not quite the level of depth or high level understand that you would want.
agreed, that's the hard part. AI helps but doesn't make it disappear. I think that history, config, tribal knowledge etc is part of the map, and you need that map to guide you when you need to dive in because your agents did something at the wrong level
I think this is probably a good way to think about it. Breadth is going to be important if you want to reasonably navigate the new world. In an emergency even if you are using an agent you will be much quicker if you know what questions to ask the agent and how to guide it.
I would argue that recognizing if the agent is an adversary also requires breadth and a willingness to dive deep when you need to. Both things that the article highlights.
Before agents that is what every high performing person I've ever worked with was like. It was the kind of person I wanted to hire when I was in charge of that sort of thing. Enough breadth to know where to dive deep and enough willingness and curiosity to dive deep.
> You need a high-level, far-away map of the entire system, and you need to know just enough to know where you need to dive in
One thing that puts me squarely in the Optimist camp is that such a map can often be created by AI just-in-time. Sure you might not know every detail of what it said, but you can simply follow up and build your understanding.
yep, the just-in-time mapping and exploration are what builds up your breadth, but I think you still need a baseline to be able to tell when the map you've been given looks right but isn't
This could be a good article but the example doesn't work?
"The CI tool was flagging an issue, perhaps correct, perhaps incorrect. But this is a real system so I had to dive in."
By diving in he ultimately just uses the AI to fix the problem:
"So I told Codex to make SonarQube happy, which it did by refactoring and splitting things up into 10 files".
Why isn't there a "shallowbot", or some CI configuration that tries to fix CI mistakes once before humans dive in? Plus it's not that deep, it's just shit code that was fixable with a "make sonarqube happy" prompt. I think the author is ashamed to admit that he's as generalist as everyone else now.
this is llm slop. and please don't flag me. this is useful information. i know hn has a policy of not commenting on asides like website css etc - but i believe ai written posts should be called out.
I like the analogy. It feels right. But I don't think it actually is, because it leans hard on the _feeling_ of failure, vs. the actual failure.
The failure that actually matters here is product failure. Angry users. Security incidents. Downtime. Broken workflows.
Stressed and uncomfortable developers have, historically, been a leading indicator of product failure. Software is complex and when devs can't handle the breadth of ownership + the depth of expertise at the same time, it means bad work that leads to product failure. It's not clear that agents fail in the same way, or that our collective intuition about how and when to trust agents is sufficiently developed to predict product failure in the same way.
As anecdata: I know a bunch of growth-SaaS/last-batch-of-IPO companies where the devs are all stressed, the feature ship rate is up, and incidents are down. All at the same time.
So yes, quick learners and just-in-time depth is important, but so is intuition about where discomfort is going to produce true failure. The discomfort isn't a feeling to throw away, simply trusting that you'll learn everything in time. Ownership requires holding a model of the system in your head and proactively steering where you dive.
I think that this was true pre-attention economy. If you look at security incidents and other prod issues, they rarely move the needle today. I work at a hyperscalar and we have been shipping 40% more bugs to prod since 2023 year-to-year, yet user growth and revenue is up about 25% similarly due to more features and more ads.
I think that the whole premise of that failure affects things is based on a older model. You kind of assume that:
1) users are able to easily gain information about alternatives. We have seen that since LLMs replaced search engines, we have much less churn.
2) users are not behaviorally addicted/pushed into patterns. We have patterns that utilize users to keep using the app by exploiting their cognitive weaknesses
3) users have the mental fortitude and stamina to learn new tools instead of accepting flaws in the one they are used to
Some failures and bugs in prod don't matters as much
Almost the entire structure of software oriented companies, and why devs are expensive, are to prevent these sorts of errors. That job has gotten harder, and in some places impossible as management forces more trust on LLMs and reduces the capacity of devs to look. This is less of a problem in existing products in production and existing cultures - revenue is protected. Eventually those structures break down, as does the user perception of quality. Once the brands have taken enough knocks, it accelerates.
I agree that not all failures matter (most don't, in fact). On the other end of the spectrum, I know of a bunch of seed/series A startups that basically operate under a model where they ship everything and fix the stuff people complain about. As long as you do that fast enough, the median user experience is pretty good, because only the first n% actually see the problem before a fix is live.
I'll go out on a limb and guess that at a hyperscaler, shipping a bug to prod means hitting canary or 1% of traffic. Seems fine!
I think my point still stands, even as the shape of failures shift. There's certain ones that still hurt (data corruption, exploited security holes, etc.) because they're hard to notice. It might even be that the primary goal of ownership is to develop intuition about where those failures are likely to happen.
There are prod issues, then there are issues that cost me money as a customer.
> Ownership requires holding a model of the system in your head and proactively steering where you dive.
Absolutely, and the feeling of discomfort is a signal that you need to dive, a signal that your mental models might need updating. I think that builds up over time through many dives
Site is down, but one thing in particular that needs close attention: db schema. I've found that LLMs make awful normalization decisions; you can limit the fallout somewhat by modularizing database access, bit it still inevitably bleeds over into the object model etc. On AI heavy projects I tend to pause feature development every few days for code & schema review, and subsequent cleanups.
Back up now, Hashnode hiccup. Agreed on schema, that's one of the places I still always dive into regardless of what the agent says
Just do the schema by hand. It forces so much downstream alignment that you can almost prefer to use the schema and any associated comments over prompting the model directly.
If you don't know what your schema should be, then I think this is the dead canary in the vibe coding coal mine. Back up to the design phase and talk to the robot about the domain until you have a really strong conviction about its tuples and relations.
This matches my own experience, pre-AI. As I've moved from IC to having a team over the years, more code was being written than I could possibly understand at the depth I would have if I'd written it myself.
The skill I learned is exactly as you describe: understand the system. For me, this means "boxes and lines", the components and how they interact. I've encouraged developers on my team to make sequence diagrams part of their deliverable, because it helps both of us put the code in the context of the system.
The boxes can be opaque because I know the boundaries and expected inputs and outputs. This tells me when I need to go in a box: is it sending a malformed message? I can trace that back into the code.
I think that's the skill we have to cultivate now that coding agents produce far more than we can review line-by-line. The system view is the map, and the boundaries tell us where to go hunting inside the box.
I've been trying to adopt this approach and struggling. Any advice on how to learn this sort of systems approach? It doesn't seem to come very naturally to me
> After formatting the files
Still shocked that people do this by hand in 2026.
> Most of the successful engineers and engineering leaders I have had the pleasure of working with were able to hold a discussion on almost any topic, whether technical or not
Reverse Gell-man? https://en.wiktionary.org/wiki/Gell-Mann_amnesia_effect#Engl...
Haha yes I've seen that too, I think the distinction is that it is not "fluent on everything" but rather "curious about everything". The best developers I know will happily admit when they don't know something and look it up, no matter the topic.
Exactly, quality thinking can apply to any level of expertise on any topic. You can be a complete novice and still have an interesting conversation
Mostly agreed with just in time learning. This is not just something that can be learned from the code though, this may require a huge amount of code context across multiple repositories and configuration files, as well as a knowledge of history, process, other tribal knowledge etc. So to me that's still the really difficult part of the job and AI definitely helps but doesn't entirely solve this. I see it often wants to solve some issue at not quite the level of depth or high level understand that you would want.
agreed, that's the hard part. AI helps but doesn't make it disappear. I think that history, config, tribal knowledge etc is part of the map, and you need that map to guide you when you need to dive in because your agents did something at the wrong level
I think this is probably a good way to think about it. Breadth is going to be important if you want to reasonably navigate the new world. In an emergency even if you are using an agent you will be much quicker if you know what questions to ask the agent and how to guide it.
yes exactly, in most cases you will have an agent you can use to help you solve the problem anyway, but you need to know what to ask it to do
Guiding works as long as the agent is neutral or on your side but if it’s secretly an adversary ..
I would argue that recognizing if the agent is an adversary also requires breadth and a willingness to dive deep when you need to. Both things that the article highlights.
Before agents that is what every high performing person I've ever worked with was like. It was the kind of person I wanted to hire when I was in charge of that sort of thing. Enough breadth to know where to dive deep and enough willingness and curiosity to dive deep.
We’ve been trusting (or distrusting) our agents for all of recorded history. The choice between Claude or Codex matters. Trust matters.
this is true for humans too
> You need a high-level, far-away map of the entire system, and you need to know just enough to know where you need to dive in
One thing that puts me squarely in the Optimist camp is that such a map can often be created by AI just-in-time. Sure you might not know every detail of what it said, but you can simply follow up and build your understanding.
yep, the just-in-time mapping and exploration are what builds up your breadth, but I think you still need a baseline to be able to tell when the map you've been given looks right but isn't
This could be a good article but the example doesn't work?
"The CI tool was flagging an issue, perhaps correct, perhaps incorrect. But this is a real system so I had to dive in."
By diving in he ultimately just uses the AI to fix the problem:
"So I told Codex to make SonarQube happy, which it did by refactoring and splitting things up into 10 files".
Why isn't there a "shallowbot", or some CI configuration that tries to fix CI mistakes once before humans dive in? Plus it's not that deep, it's just shit code that was fixable with a "make sonarqube happy" prompt. I think the author is ashamed to admit that he's as generalist as everyone else now.
this is llm slop. and please don't flag me. this is useful information. i know hn has a policy of not commenting on asides like website css etc - but i believe ai written posts should be called out.
TLDR AI can write the code, but you still need enough technical knowledge to recognise bad decisions and fix problems.