Coordinators and other agents are as unreliable as other agents. It has the same problems, empty assertions, undiscovered mismatches, and outdated memories. I tend to work with multiple agents in parallel. Based on the same agreement, everyone chooses a more suitable agent to do the final finishing when they need to make decisions.
This is close to my default workflow. One Fable to rule them all, many Opuses to implement. All work is planned, tracked and logged in plain Markdown tickets. No magic. Nothing to talk about.
Still, I have to keep an eye on it or it devolves into a mess real fast. An example from yesterday: I noticed the commit added 10K lines where it clearly should not have. 5min of digging: it was a combinatorial state explosion in the parsers. All at once because a shared grammar was tweaked.
Oopsies like this happen all the time, and with every passing hour it becomes harder to fix.
Can only rely on that chief-of-stuff for mundane things.
i mean its not wrong, basically everyone learns this within a few weeks of actively working with the agentic loop.
but as usual with ai written content, the word bloat is roughly x5 of the words necessary to convey the message - with basically no effort on the meat proxy's part to clean it up in any way, shape of form
Perhaps I'm just "stuck in the past" but I do not understand the appeal of working like this. Your app is being built on architectural quicksand.
I just work on one thing at a time, always with Plan mode upfront, and I'd say most of the time I have some feedback to refine the plan. Working good so far.
i've been working on my own toy "software factory" concept (roughly 300k backend and 100k frontend loc across all components of the platform as of today).
I've been building it to ultimately run it on my homeserver - its strongly focused on maintenance tasks like automatic library updates and rebuilding base images for deployment (and actual deployment), etc pp.
My explicit goal of the project was to replace my currently manually managed TrueNAS Scale installation, on which i'm currently (more or less) manually managing the lifecycle of various self hosted applications.
So i'm expecting the factory to eg versionbump dependencies, read changelogs, copy the data of the selfhosted applications to prerun the migrations and verify the migration does not encounter issues. Or similarly for my own written software, eg if the framework i'm using in a project has released a new major version... same route, read changelogs, trial migration etc.
While I would be able to build such a system on my own, realistically speaking I would never because the needed time to actually do it far exceeds the amount of time I have available next to my full-time dev job.
So I've basically decided to go with the llm-driven development flow, where I still have a mental model of the system to the bone/internals, and explicitly state how it should be developed. It usually takes roughly 15-30 minutes to properly scope a development, which then takes agents 4+ hours to implement and fully e2e test on the dev platform.
eg. yesterday evening i decided to merge the Ticket and Epic/Story/Task concepts into a unified model / same entity table. I scoped it yesterday afternoon, sent of the process to develop it in the evening and it got finished around 4am this morning. (And it totally wasted multiple hours waiting on events to occur which it filtered out via a shell pipe, sigh)
My mindset building it is basically the same as i tread a Factorio, DSP or Satisfactory game: there is some jank to it for sure, but if the process keeps going, its fine. And its kinda fun to see things getting more and more streamlined over the months.
I dont think i'd be comfortable treating my dayjobs codebase like that, however.
To a large part because i'm missing essential QA that I automated in my own project, eg you cannot have any unattended mutations unless theyre easily revertable. At my dayjob, if I merge and deploy a faulty commit, i _will_ impact others. When the same happens on my personal project ... some agents may be inconvenienced, which is not particularly high on my priority list.
Bitter lesson means all these tricks will not be needed in a year or two. Either labs will abstract it in harness or models will become good enough that it can do it by itself
I disagree. Parallelization, coordination shouldn't be model-level functionality. Further, how do you know if or when we'll get orders of magnitude increases in context sizes. Or, do you mean that the labs will just nail the perfect plug-and-play harness? That's fair, but why wait.
If implemented correctly these tricks will still work on less expensive models delivering nearly identical value.
Today, a hefty amount of standard coding tasks can be completed with similar results to gpt astra using terra and a tailored harness around it.
Also the scale matters. One big, expensive session, with a frontier model paired with a dev-babysitter is ok. But make it a factory (kindergarden: few devs, many parallel streams) and you'll want to follow a strict protocol.
Models are already good enough. Last week I had Fable plan out a project that took approx 4 days end to end with each phase orchestrated by a supervisor agent delegating individual tasks to other agents, coordinating everything and checking status by simple text files in the repo. I didn’t have to tell the agent to do it that way, it just came up with it and set up the infra as part of the planning overview. Great that everyone’s posting their “my secret sauce” cookbooks just to jump on the hype train but it looks like the models have already figured it out for themselves.
What is the interplay between such delegation and token cache timeout? If Fable is truly active for 4 days, enough to keep the cache hot, the cost would be astronomical. OTOH, if Fable idles while subagents are active, then each awakening is a cache miss.
True, that’s why I didn’t have a single Fable thread running for the whole time. Each phase had its own orchestrator that ran about 4-5h, in Fable or Opus depending on complexity, delegating to sub agents along the way - they had tasks broken down in a granular enough way that each returned in anything from 5 to 30 mins approx, then the lead agent did some validating and status updates before delegating the next step. So the main agent’s cache never went cold. Each phase raised its own PR and had an opportunity for human validation/code review before the next phase kicked off in a new context.
> Great that everyone’s posting their “my secret sauce” cookbooks just to jump on the hype train but it looks like the models have already figured it out for themselves
Unfortunate lesson to be learned here: there's not much leverage here other than just using AI. Previously, us devs could get a head start and build some institutional knowledge but not this time. I'm bearish on all the custom harnesses stuff that people talk about.
What helps me is to understand the failure modes of LLMs - it can't be articulated in easy words but something you can learn slightly by just using it. For example I have an intuition of when to start compacting but Codex already does it for you now haha.
My take: the highest leverage move for us is to write AGENTS.md and provide everything that the model can't learn on its own or might take time to learn.
Yes it’ll eventually get built into the harness/model/interface; just like how “work it out step by step” became thinking, but if you have work to do today, it can be worth trying to improve it.
I’ll tell the missing part, what comes next: the team, with teammates and managers, starts making their product decisions. If the user was anywhere near the chat and wrote a mere “ok”, they will be recorded as “user's”.
They will write tests. Lots of tests. Instead of removing any code, there will be 3 layers of backward compatibility, and tests that test presence of tests that test that backward compatibility.
The reviews will find all possible edge cases, including those that can never happen, and make the UI gracefully handle them. With tests.
The diff from any integration PR from team work will be over 10K lines, half of them bureaucracy. Zero chance to review even one – they will churn half-a-dozen per day.
For anything outside of known shape, the original hard topics become quickly displaced with shortcuts and familiar patterns.
Next, the app will break under load, and you will find that it’s caused by a quadratic sweep over the whole DB on any insert to prevent something irrelevant that you specifically told not to do.
You will ask, “wtf? why is it there?”. “It’s load-bearing, you ruled it”.
(That's not a joke. That's how I spent the summer.)
It begins as “let’s give the agents a kanban board to track stuff”. Then, why not have one agent do the tracking while another does the development?
Then why not have a fleet of them - specialists doing their thing? Then why not have them communicate in a standard way? Then, oh now we have so much docs/messages that we are getting lost. Why not add memory and semantic search for the project?
The rabbit hole keeps going until you run the project and find silly stupid logical issues and wonder “is this what I burnt all those tokens for?! Why is it so over-engineered?!”
I have one agent now that I use to fill in at specific places in functions/modules that I have created and working on.
Slow down and use AI to just do the tightly scoped mundane work. It is nice and effective. We all don’t need to save the world.
As someone who's been trying to build a single project since February, I never expected things to take this long. "The team" keeps finding things to extend because of some issue somewhere (so many race conditions), and I'm constantly worried that things wouldn't be good enough to launch.
I've started adding a "let's make sure we avoid overengineering" to my plan review prompts. I'm still hoping I can launch in maybe two weeks.
I tried this approach and the agents just built tons of tests. The Agent in Charge ordered more and more tests. After two weeks it reported finished” and the end product was completely unusable.
I really hope future models will do a better job at this. As it works be useful (if it worked).
Coordinators and other agents are as unreliable as other agents. It has the same problems, empty assertions, undiscovered mismatches, and outdated memories. I tend to work with multiple agents in parallel. Based on the same agreement, everyone chooses a more suitable agent to do the final finishing when they need to make decisions.
This is close to my default workflow. One Fable to rule them all, many Opuses to implement. All work is planned, tracked and logged in plain Markdown tickets. No magic. Nothing to talk about.
Still, I have to keep an eye on it or it devolves into a mess real fast. An example from yesterday: I noticed the commit added 10K lines where it clearly should not have. 5min of digging: it was a combinatorial state explosion in the parsers. All at once because a shared grammar was tweaked. Oopsies like this happen all the time, and with every passing hour it becomes harder to fix.
Can only rely on that chief-of-stuff for mundane things.
> What this is normally called
Is this also what Microsoft calls "Magentic" pattern?
(for a long time i kept reading it as Magnetic pattern)
https://learn.microsoft.com/en-us/semantic-kernel/frameworks...
i mean its not wrong, basically everyone learns this within a few weeks of actively working with the agentic loop.
but as usual with ai written content, the word bloat is roughly x5 of the words necessary to convey the message - with basically no effort on the meat proxy's part to clean it up in any way, shape of form
Perhaps I'm just "stuck in the past" but I do not understand the appeal of working like this. Your app is being built on architectural quicksand.
I just work on one thing at a time, always with Plan mode upfront, and I'd say most of the time I have some feedback to refine the plan. Working good so far.
As you get better with the pattern, you can do much more while maintaining the quality.
Don't you end up sitting there waiting while the agent goes off and executes the plan?
No I do other things. Oftentimes I'll just do it from my phone when I'm out and about and just work with it asynchronously.
eh, it has its own charm.
i've been working on my own toy "software factory" concept (roughly 300k backend and 100k frontend loc across all components of the platform as of today). I've been building it to ultimately run it on my homeserver - its strongly focused on maintenance tasks like automatic library updates and rebuilding base images for deployment (and actual deployment), etc pp.
My explicit goal of the project was to replace my currently manually managed TrueNAS Scale installation, on which i'm currently (more or less) manually managing the lifecycle of various self hosted applications.
So i'm expecting the factory to eg versionbump dependencies, read changelogs, copy the data of the selfhosted applications to prerun the migrations and verify the migration does not encounter issues. Or similarly for my own written software, eg if the framework i'm using in a project has released a new major version... same route, read changelogs, trial migration etc.
While I would be able to build such a system on my own, realistically speaking I would never because the needed time to actually do it far exceeds the amount of time I have available next to my full-time dev job.
So I've basically decided to go with the llm-driven development flow, where I still have a mental model of the system to the bone/internals, and explicitly state how it should be developed. It usually takes roughly 15-30 minutes to properly scope a development, which then takes agents 4+ hours to implement and fully e2e test on the dev platform. eg. yesterday evening i decided to merge the Ticket and Epic/Story/Task concepts into a unified model / same entity table. I scoped it yesterday afternoon, sent of the process to develop it in the evening and it got finished around 4am this morning. (And it totally wasted multiple hours waiting on events to occur which it filtered out via a shell pipe, sigh)
My mindset building it is basically the same as i tread a Factorio, DSP or Satisfactory game: there is some jank to it for sure, but if the process keeps going, its fine. And its kinda fun to see things getting more and more streamlined over the months.
I dont think i'd be comfortable treating my dayjobs codebase like that, however. To a large part because i'm missing essential QA that I automated in my own project, eg you cannot have any unattended mutations unless theyre easily revertable. At my dayjob, if I merge and deploy a faulty commit, i _will_ impact others. When the same happens on my personal project ... some agents may be inconvenienced, which is not particularly high on my priority list.
Quality is in the economy I guess.
Bitter lesson means all these tricks will not be needed in a year or two. Either labs will abstract it in harness or models will become good enough that it can do it by itself
I disagree. Parallelization, coordination shouldn't be model-level functionality. Further, how do you know if or when we'll get orders of magnitude increases in context sizes. Or, do you mean that the labs will just nail the perfect plug-and-play harness? That's fair, but why wait.
If implemented correctly these tricks will still work on less expensive models delivering nearly identical value.
Today, a hefty amount of standard coding tasks can be completed with similar results to gpt astra using terra and a tailored harness around it.
Also the scale matters. One big, expensive session, with a frontier model paired with a dev-babysitter is ok. But make it a factory (kindergarden: few devs, many parallel streams) and you'll want to follow a strict protocol.
Models are already good enough. Last week I had Fable plan out a project that took approx 4 days end to end with each phase orchestrated by a supervisor agent delegating individual tasks to other agents, coordinating everything and checking status by simple text files in the repo. I didn’t have to tell the agent to do it that way, it just came up with it and set up the infra as part of the planning overview. Great that everyone’s posting their “my secret sauce” cookbooks just to jump on the hype train but it looks like the models have already figured it out for themselves.
What is the interplay between such delegation and token cache timeout? If Fable is truly active for 4 days, enough to keep the cache hot, the cost would be astronomical. OTOH, if Fable idles while subagents are active, then each awakening is a cache miss.
True, that’s why I didn’t have a single Fable thread running for the whole time. Each phase had its own orchestrator that ran about 4-5h, in Fable or Opus depending on complexity, delegating to sub agents along the way - they had tasks broken down in a granular enough way that each returned in anything from 5 to 30 mins approx, then the lead agent did some validating and status updates before delegating the next step. So the main agent’s cache never went cold. Each phase raised its own PR and had an opportunity for human validation/code review before the next phase kicked off in a new context.
true haha
> Great that everyone’s posting their “my secret sauce” cookbooks just to jump on the hype train but it looks like the models have already figured it out for themselves
Unfortunate lesson to be learned here: there's not much leverage here other than just using AI. Previously, us devs could get a head start and build some institutional knowledge but not this time. I'm bearish on all the custom harnesses stuff that people talk about.
What helps me is to understand the failure modes of LLMs - it can't be articulated in easy words but something you can learn slightly by just using it. For example I have an intuition of when to start compacting but Codex already does it for you now haha.
My take: the highest leverage move for us is to write AGENTS.md and provide everything that the model can't learn on its own or might take time to learn.
So we should just ignore?
If you’re using models today, it’s worth it.
Yes it’ll eventually get built into the harness/model/interface; just like how “work it out step by step” became thinking, but if you have work to do today, it can be worth trying to improve it.
I’ll tell the missing part, what comes next: the team, with teammates and managers, starts making their product decisions. If the user was anywhere near the chat and wrote a mere “ok”, they will be recorded as “user's”.
They will write tests. Lots of tests. Instead of removing any code, there will be 3 layers of backward compatibility, and tests that test presence of tests that test that backward compatibility.
The reviews will find all possible edge cases, including those that can never happen, and make the UI gracefully handle them. With tests.
The diff from any integration PR from team work will be over 10K lines, half of them bureaucracy. Zero chance to review even one – they will churn half-a-dozen per day.
For anything outside of known shape, the original hard topics become quickly displaced with shortcuts and familiar patterns.
Next, the app will break under load, and you will find that it’s caused by a quadratic sweep over the whole DB on any insert to prevent something irrelevant that you specifically told not to do.
You will ask, “wtf? why is it there?”. “It’s load-bearing, you ruled it”.
(That's not a joke. That's how I spent the summer.)
+1 my last 2 months….
It begins as “let’s give the agents a kanban board to track stuff”. Then, why not have one agent do the tracking while another does the development? Then why not have a fleet of them - specialists doing their thing? Then why not have them communicate in a standard way? Then, oh now we have so much docs/messages that we are getting lost. Why not add memory and semantic search for the project?
The rabbit hole keeps going until you run the project and find silly stupid logical issues and wonder “is this what I burnt all those tokens for?! Why is it so over-engineered?!”
I have one agent now that I use to fill in at specific places in functions/modules that I have created and working on.
Slow down and use AI to just do the tightly scoped mundane work. It is nice and effective. We all don’t need to save the world.
As someone who's been trying to build a single project since February, I never expected things to take this long. "The team" keeps finding things to extend because of some issue somewhere (so many race conditions), and I'm constantly worried that things wouldn't be good enough to launch.
I've started adding a "let's make sure we avoid overengineering" to my plan review prompts. I'm still hoping I can launch in maybe two weeks.
Lol, this was my summer too!
I tried this approach and the agents just built tons of tests. The Agent in Charge ordered more and more tests. After two weeks it reported finished” and the end product was completely unusable.
I really hope future models will do a better job at this. As it works be useful (if it worked).
[flagged]