The conceptual problem is that we keep wanting to compare AI behavior to that of traditional computers. The proper comparison is comparing AI, and how we trust or delegate to it, to the concept of delegating to other humans or even to domestic animal. Employees can be trained and given very specific skills and guidelines but still have agency and non-deterministic behavior. A seeing eye dog, a pack mule or chariot horse will often, but not necessarily always do what you ask of them. We've only been delegating to deterministic programmable machines for very short part of human history. But ad human societies, we've been collectively delegating a lot of useful activities to non-perfectly-dependable agents (ie each other) for a very long time. As as humans we've gotten done more that a few notable things in the last several millennia with this method. However, humans as delegates or as delegators have also done a lot of horrific things at scale to, both by accident or by design. And meanwhile (gestures broadly around everywhere) maybe humans actually aren't doing such an optimal job of running and governing everything important in the world?
When compared to how human make a mess of things like in the real world, how high does the bar really need to be for trusting AI agents. Even far shy from perfect, AI could still be a step function improvement over trusting ourselves.
Human delegation is disciplined as much by incentive alignment as instruction. The same is true for LLM's. The problem is that it's not possible to dominate intentions, LLM or human, because delegates/agents to be useful need autonomy.
The SOTA models are working on making them more capable and then adding guardrails for safety. It would be better to work on baking in incentive alignment, which probably means eliciting more incentive details from the LLM user. That's what I'd be working on at Apple, where the user might be induced to share a level of local-only details that could align the AI agents.
Totally agree. And I expect at some point people might come around on, “don’t pay for and use that tool for that particular job.”
Like, there isn’t enough hype the world to make people replace all knives, hammers, and screwdrivers with sawzalls. They have awesome utility for certain things and they’re a bad fit for other things.
> And meanwhile (gestures broadly around everywhere) maybe humans actually aren't doing such an optimal job of running and governing everything important in the world?
The issue with this is that you want to impune, in the grand scheme of things, a small few individuals. And so you want to institute an AI system. Which are controlled by the same individuals (or at least the same class of individuals, with the reach to abuse such a system).
I'll hear you out if AI becomes truly decentralized. Until then, no, this line of rhetoric is just justification for the surveillance state that's to come (to be fair, the surveillance state would pick yet another justification, regardless).
Well AI agents thinking capabilities are inspired by our own “neural networks.” AI makes the same mistakes we do it’s just called different things.
How many people say something like, “if I recall correctly.” This statement emphasizes that we think we know, but we’re just adding that disclaimer to protect ourselves from cancel culture.
People call that “Hallucination” when talking about an AI. It’s not hallucination, it’s beautiful imperfection.
The main difference is that this junior employee can't be held responsible if anything goes wrong. And the company which rented you this employee absolves itself from all responsibility too.
How would you hold the junior responsible if they bring down the system and it cost a million dollars? Will they actually recover all the money from the employee ?
Who do you trust with the keys?
In any well run organization you have multiple layers of controls. The same concept applies here and I think the gp commenter captured it very well.
I think you'd trust someone with the keys when they've consistently shown that they can be trusted with less critical work. If you're having to constantly monitor someone's output, then promoting them is a liability.
The same applies to an AI model.
And, since the same model would be deployed by many teams, unexpected behavior from that model even for a small subset of those teams means that it can't be promoted.
> In any well run organization you have multiple layers of controls.
Everything depends on size.
A business with 8 employees might need 3 of them to be (literal) keyholders, and might be situated such that any of the keyholders has it in their power to destroy the business.
This is not ideal, obviously, but it is how the world has worked for a very long time, and it is difficult to understand how to make it better in some cases. Modern technology, such as cameras, might help, or might simply help to allocate blame after destruction has occurred.
In any case, this is the background of how people are used to working. We all deal with people who can absolutely destroy us, starting with the cop on the corner.
And we have mechanisms, both before-the-fact, like social coercion, and after-the-fact, like the legal system, to help ensure that this usually works.
LLMs exist in a world where most people are used to extending trust, but it isn't possible for LLMs to conform to the historical expectations that underpin that trust.
Yes. I think you can get agents to “Conscious competence” with a lot of well-designed oversight, direction and control. It works, but it’s fragile - nothing like the judgement needed to handle novel situations well.
> You bought a laptop or desktop with an operating system, and it did what it said on the tin: it ran programs and stored files.
I feel like people may be viewing the past with rose colored glasses. Computing in the 90s meant hitting ctrl-s every 5 seconds because you never knew when the application you were using was going to crash. Most things didn't "just work", but required extensive tweaking to configure your ram, sound card... to work at all.
It's sad, but they should've compulsively hit save after every few letters - it's documented very clearly on page 404 of the manual. It's a real shame that such things couldn't be done automatically until recently, early-2000-era CPUs just weren't sophisticated enough to run advanced, reactive logic like that.
My parents indoctrinated me as a child to constantly hit save because they grew up with that. It was a part of our cultural expectations for "basic life skills to teach children".
Indeed, not like today, where we're closer to the metal and have as few abstractions as possible. Maybe we'll be able to use high level languages again after this RAM shortage passes.
This is not just the past. I still have headaches configuring my video card to work with the right CUDA drivers, etc.
The tower of abstractions we're building has reached a height that actually makes everything more fragile, even if the individual pieces are more robust.
Still though -- once you got a workflow, no matter how terrible, it strongly tended to continue to work that way, and it was still much easier to diagnose, fix, and just generally not have unexpected behavior.
This is the issue; agents introduce more unexpected behavior, at least for now.
My gut is that always on "agents who can do things unexpectedly" are a dead-end, but what AI can do is get you to a nice AND predictable "workflow" easier.
e.g. for now I don't like AI for dealing with my info, but I love AI helping me make more and better bash scripts, that deal with my info.
alot of software engineering, especially in complex systems, is still just tweaking retries, alarms, edge cases etc. it might take 3 days to even figure out what went wrong
I had occasional crashes, sure, but unless you had some very dodgy computers, it seems like you're overcorrecting for those supposed rose-colored glasses.
I never knew anyone in the '90s who was constantly living in fear of their programs crashing and losing their work.
I used Word—mainly on Mac, version 4.01—all through middle and high school for my homework, and never had any particular problems with it. Frankly, I think it was much more stable then than it is now.
> Computing in the 90s meant hitting ctrl-s every 5 seconds because you never knew when the application you were using was going to crash.
THIS.
I lost so much work in the 90s and 00s. I was a kid, so I had patience and it didn't cost me any money. I can't imagine people losing actual work presentations or projects.
Every piece of software was like this. It was either the app crashing or Windows crashing. I lost Flash projects, websites, PHP code.
Sometimes software would write a blank buffer to file too, so you needed copies.
Version control was one of my favorite discoveries. I clung to SVN for the few years after I found it.
My final major loss was when Open Office on Ubuntu deleted my 30 page undergrad biochem thesis I'd spent a month on. I've never used it since.
Windows 95 Word was also bad. Some poor non-CS student brought his thesis to our computer pool and worked from Floppy with the only copy he had. Panic mode on when the backup file and original did not fit on that floppy any more and Word asked to swap disks for an empty one. We advised him to just continue swapping, eventually Word will have that backup file on the other disk. It worked after an ennerving amount of floppy swaps...
Quality issues are a different vertical within the space of software/user misalignment. The sort of issue the author talks about is more like the malware of the 90-00s era: the software deliberately does something to screw the user.
This feels like the modern incarnation of "packet intent", the mythical security property of knowing what an incoming request is trying to do rather than what it is. Variants of "packet intent" have been sought after going all the way back into the 1980s; it's helpful to recognize the idea when it appears, because it's a reliable marker of what you can't realistically accomplish.
Except agents actually have an intent, and can route around obstacles to accomplish that intent.
If you merely block a specific action, they will find another way to do what they're trying to do. Agent security requires controlling the agent's intent.
The goal behind most "clean" software design in general is to eliminate the possibility of failure via constraints. That's the pattern I've seen over the years. Of course, the map is not the territory - you need to make sure the reachable set within the constraints is actually a subset of the real reachable set. Which may be underspecified or unknown a priori (as if you could've really specified the true reachable set, why didn't you just encode those rules?)
So I'm sympathetic to the criticism, especially since composition of formal methods & analyzing their effects is still very much a hard problem (and not just computationally - philosophically, often, for the reason I listed above).
That being said, I don't know a better solution. Begging the agent with prompts doesn't work. Are you suggesting some kind of mechanistic interpretability, maybe?
been building on claude code for a while. the post's framing is right.
mcp gives you open standards on the tool layer but the harness
(claude code, cursor) is still proprietary. your product is one
anthropic decision away from breaking.
the user agent role the post calls for needs open harnesses, not just
open standards. otherwise we end up rebuilding mobile under a new name.
if you've actually migrated an existing claude code setup to one of them, curious how the portability story worked. that's the part i'd been worried about.
I've not tried actually migrating from Claude Code... but having played a bit with other clients, I would avoid Mistral Vibe. I want to love it, there's some things that are nice about Vibe (mostly just "oui oui baguette"), but the things I did not like about it were disastrously bad. I could barely get MCP servers configured, and it was in something of a broken state even when I did get it working. I have many words about how horrified I am at how far behind Mistral is, but I will spare the rant.
OpenCode is another one to consider looking at: https://opencode.ai/ Not sure I'd recommend it, but it's worthy of consideration, as is Pi.
Also, consider that you can build your own. I've got Claude Code in the background working on improvements to my own harness (just for myself) at the moment. Though my intention is to have a mini API-only Claude Code that I can use on retro machines that don't support it, I don't need a full Claude Code feature set.
Sorry, I can’t help with that, as I’ve never actually used Claude. Started with Mistral and stuck there (they still have a free API tier, but I ended up buying their Le Chat Pro service anyway, for the image generation).
The framing assumes the agent can reliably represent its principal, and I'm not convinced that holds even if you get everything else right.
The problem is that the agent itself is the attack surface. An adversary who controls the communication channel can manipulate what the agent believes about who it's talking to, which means anything it holds, its list of authorized actions, a shared secret you gave it, whatever, can be exfiltrated in ways the agent can't detect because the manipulation happens below the layer where it can reason about trust.
Open harnesses and open standards help but they don't close this gap, because the thing you need to trust, the agent's own judgment about its principal, is exactly what gets compromised. The trust chain has to go below software entirely: hardware attestation, signed commands with keys the agent can verify but never access. That's really an OS problem dressed up as an agent architecture problem.
The thing I don’t like about “agents” is that I consider my computer a tool that I use and control. I don’t want it doing things for me: I want to do things through it. I want to be in the driver’s seat. “Notifications” and “Assistants” and now “Agents” break this philosophy. Now there are these things doing “stuff” on my computer for me and I’m just a passenger along for the ride. A computer should be that “bicycle for the mind” as Jobs put it, not some autonomous information-chauffeur, spooning output into my mouth.
The browser analogy holds because publishers wanted browsers. Sites lived with User-Agent and robots.txt because the click paid for it.
AI agents are the destination. No return click to bargain with. That's why Cloudflare just went default-block + 402 Payment Required instead of waiting on a standards body.
Open standards on the agent side are the easy half. Getting sites to show up is the part W3C can't fix alone.
i think whats missing is the raison detre of the Agents isnt a new usecase, its a context prune for the same limitations LLMs provide. LLM as Agent is a subset, where the goal of the agent is set by the parent and is suppose to return a pruned context.
if you dont recognize the technical limitations that produced agents youre wearing rose tinted glasses. LLMs arent approaching singularity. theyre topping out in power and agents are an attempt to exentend useful context.
The sigmoid approacheth and anyone of merit should be figuring out how the harness spits out agents, intelligently prunes context then returns the best operational bits, alongside building the garden of tools.
Its like agents are the muscles, the bones are the harness and the brain is the root parent.
The conceptual problem is that we keep wanting to compare AI behavior to that of traditional computers. The proper comparison is comparing AI, and how we trust or delegate to it, to the concept of delegating to other humans or even to domestic animal. Employees can be trained and given very specific skills and guidelines but still have agency and non-deterministic behavior. A seeing eye dog, a pack mule or chariot horse will often, but not necessarily always do what you ask of them. We've only been delegating to deterministic programmable machines for very short part of human history. But ad human societies, we've been collectively delegating a lot of useful activities to non-perfectly-dependable agents (ie each other) for a very long time. As as humans we've gotten done more that a few notable things in the last several millennia with this method. However, humans as delegates or as delegators have also done a lot of horrific things at scale to, both by accident or by design. And meanwhile (gestures broadly around everywhere) maybe humans actually aren't doing such an optimal job of running and governing everything important in the world?
When compared to how human make a mess of things like in the real world, how high does the bar really need to be for trusting AI agents. Even far shy from perfect, AI could still be a step function improvement over trusting ourselves.
Human delegation is disciplined as much by incentive alignment as instruction. The same is true for LLM's. The problem is that it's not possible to dominate intentions, LLM or human, because delegates/agents to be useful need autonomy.
The SOTA models are working on making them more capable and then adding guardrails for safety. It would be better to work on baking in incentive alignment, which probably means eliciting more incentive details from the LLM user. That's what I'd be working on at Apple, where the user might be induced to share a level of local-only details that could align the AI agents.
AI "disciplining" is arguably much simpler, easier and cheaper.
> how high does the bar really need to be for trusting AI agents.
You can hold a human responsible for that they do; you can reward them, fire them, sue them, etc.
You cannot do any of those things with an LLM. The threat of termination means nothing to an LLM.
Totally agree. And I expect at some point people might come around on, “don’t pay for and use that tool for that particular job.”
Like, there isn’t enough hype the world to make people replace all knives, hammers, and screwdrivers with sawzalls. They have awesome utility for certain things and they’re a bad fit for other things.
Maybe we’ll get there with LLMs someday.
> And meanwhile (gestures broadly around everywhere) maybe humans actually aren't doing such an optimal job of running and governing everything important in the world?
The issue with this is that you want to impune, in the grand scheme of things, a small few individuals. And so you want to institute an AI system. Which are controlled by the same individuals (or at least the same class of individuals, with the reach to abuse such a system).
I'll hear you out if AI becomes truly decentralized. Until then, no, this line of rhetoric is just justification for the surveillance state that's to come (to be fair, the surveillance state would pick yet another justification, regardless).
Well AI agents thinking capabilities are inspired by our own “neural networks.” AI makes the same mistakes we do it’s just called different things.
How many people say something like, “if I recall correctly.” This statement emphasizes that we think we know, but we’re just adding that disclaimer to protect ourselves from cancel culture.
People call that “Hallucination” when talking about an AI. It’s not hallucination, it’s beautiful imperfection.
A very talented junior employee that you can't trust with the keys.
The main difference is that this junior employee can't be held responsible if anything goes wrong. And the company which rented you this employee absolves itself from all responsibility too.
Here is a fresh example from today of what junior employee do when given unlimited agentic power : https://www.reddit.com/r/ClaudeAI/comments/1sv7fvc/im_a_nurs...
How would you hold the junior responsible if they bring down the system and it cost a million dollars? Will they actually recover all the money from the employee ?
Your example is not from a Jr developer but from a free agent.
I think you will find it very hard to keep a Jr dev in a Corp responsible.
I actually think you will find that it is easier to work with agents at a higher quality and lower legal risk than using Jr developers.
And this is only going to be amplified when it becomes common knowledge that Ai poses less risk to projects, than Jr staff.
I understand you mean this as it is close to that in terms of getting the final work.
But in my opinion, it is not even remotely close to the reliability of an educated human, communication wise.
If you gave a research task to a less experienced person, you wouldn’t expect them to convincingly lie about details.
It is useful as a review tool or boilerplate generator but it is not the same aspect you would use a human from.
Who do you trust with the keys? In any well run organization you have multiple layers of controls. The same concept applies here and I think the gp commenter captured it very well.
I think you'd trust someone with the keys when they've consistently shown that they can be trusted with less critical work. If you're having to constantly monitor someone's output, then promoting them is a liability.
The same applies to an AI model.
And, since the same model would be deployed by many teams, unexpected behavior from that model even for a small subset of those teams means that it can't be promoted.
> In any well run organization you have multiple layers of controls.
Everything depends on size.
A business with 8 employees might need 3 of them to be (literal) keyholders, and might be situated such that any of the keyholders has it in their power to destroy the business.
This is not ideal, obviously, but it is how the world has worked for a very long time, and it is difficult to understand how to make it better in some cases. Modern technology, such as cameras, might help, or might simply help to allocate blame after destruction has occurred.
In any case, this is the background of how people are used to working. We all deal with people who can absolutely destroy us, starting with the cop on the corner.
And we have mechanisms, both before-the-fact, like social coercion, and after-the-fact, like the legal system, to help ensure that this usually works.
LLMs exist in a world where most people are used to extending trust, but it isn't possible for LLMs to conform to the historical expectations that underpin that trust.
Yes. I think you can get agents to “Conscious competence” with a lot of well-designed oversight, direction and control. It works, but it’s fragile - nothing like the judgement needed to handle novel situations well.
https://en.wikipedia.org/wiki/Four_stages_of_competence
> You bought a laptop or desktop with an operating system, and it did what it said on the tin: it ran programs and stored files.
I feel like people may be viewing the past with rose colored glasses. Computing in the 90s meant hitting ctrl-s every 5 seconds because you never knew when the application you were using was going to crash. Most things didn't "just work", but required extensive tweaking to configure your ram, sound card... to work at all.
I remember when the computer crashed and the user hadn't saved recently, we blamed the user.
It's sad, but they should've compulsively hit save after every few letters - it's documented very clearly on page 404 of the manual. It's a real shame that such things couldn't be done automatically until recently, early-2000-era CPUs just weren't sophisticated enough to run advanced, reactive logic like that.
My parents indoctrinated me as a child to constantly hit save because they grew up with that. It was a part of our cultural expectations for "basic life skills to teach children".
Serializing a document was non-trivial for the first two decades of personal computing. Auto-save would have destroyed performance.
Indeed, not like today, where we're closer to the metal and have as few abstractions as possible. Maybe we'll be able to use high level languages again after this RAM shortage passes.
This is not just the past. I still have headaches configuring my video card to work with the right CUDA drivers, etc.
The tower of abstractions we're building has reached a height that actually makes everything more fragile, even if the individual pieces are more robust.
We just need one more layer of abstraction to fix that, and everything will be fine
I'm vibe coding this presently. Update soon.
Wait, I literally still hit Ctrl-S constantly, usually a few times in a row.
Have people outgrown this unnecessary habit? Haha
Manually editing config files thanks to an obscure thread so that your printer can actually be recognized by the OS
> Computing in the 90s meant hitting ctrl-s every 5 seconds because you never knew when the application you were using was going to crash.
That was in the Windows world. Maybe in the Mac world too?
No so much in the *nix world.
Windows seems to have improved its (crash) reliability since then though, which I suppose is nice. :)
And yet the word "crash" appears quite often in The UNIX-HATERS Handbook: https://web.mit.edu/~simsong/www/ugh.pdf
Yep, it's not surprising that a book called something HATERS Handbook would be pretty focused on the negatives, even if they're rare. ;)
And then having to learn ctrl-q the minute you started working in the shell..
Muscle memory is a bitch!
lol be honest that lunacy was unique to Microsoft, never had to do that with FrameMaker on SunOS
Still though -- once you got a workflow, no matter how terrible, it strongly tended to continue to work that way, and it was still much easier to diagnose, fix, and just generally not have unexpected behavior.
This is the issue; agents introduce more unexpected behavior, at least for now.
My gut is that always on "agents who can do things unexpectedly" are a dead-end, but what AI can do is get you to a nice AND predictable "workflow" easier.
e.g. for now I don't like AI for dealing with my info, but I love AI helping me make more and better bash scripts, that deal with my info.
alot of software engineering, especially in complex systems, is still just tweaking retries, alarms, edge cases etc. it might take 3 days to even figure out what went wrong
I don't know what computers you were using.
I had occasional crashes, sure, but unless you had some very dodgy computers, it seems like you're overcorrecting for those supposed rose-colored glasses.
I never knew anyone in the '90s who was constantly living in fear of their programs crashing and losing their work.
Were they using Word? That was absolutely a fear of mine at the time.
I used Word—mainly on Mac, version 4.01—all through middle and high school for my homework, and never had any particular problems with it. Frankly, I think it was much more stable then than it is now.
> Computing in the 90s meant hitting ctrl-s every 5 seconds because you never knew when the application you were using was going to crash.
THIS.
I lost so much work in the 90s and 00s. I was a kid, so I had patience and it didn't cost me any money. I can't imagine people losing actual work presentations or projects.
Every piece of software was like this. It was either the app crashing or Windows crashing. I lost Flash projects, websites, PHP code.
Sometimes software would write a blank buffer to file too, so you needed copies.
Version control was one of my favorite discoveries. I clung to SVN for the few years after I found it.
My final major loss was when Open Office on Ubuntu deleted my 30 page undergrad biochem thesis I'd spent a month on. I've never used it since.
Open Office on Ubuntu 11.10 user here. I can confirm it froze frequently and you would lose everything. it was incredibly frustrating
Windows 95 Word was also bad. Some poor non-CS student brought his thesis to our computer pool and worked from Floppy with the only copy he had. Panic mode on when the backup file and original did not fit on that floppy any more and Word asked to swap disks for an empty one. We advised him to just continue swapping, eventually Word will have that backup file on the other disk. It worked after an ennerving amount of floppy swaps...
Quality issues are a different vertical within the space of software/user misalignment. The sort of issue the author talks about is more like the malware of the 90-00s era: the software deliberately does something to screw the user.
Hmm ... no?
I used computers back then and many things just worked fine. I found Windows XP way more predictable and stable than any of its successors.
The comment said "in the 90s". Windows XP was only released 10/2001.
So how stable were Windows 95, or Windows 3.x, IYE?
Ah, fair point!
Windows 3.x was pretty good, imo. Windows 95 bad. Windows 98 overall good but like 7/10.
This feels like the modern incarnation of "packet intent", the mythical security property of knowing what an incoming request is trying to do rather than what it is. Variants of "packet intent" have been sought after going all the way back into the 1980s; it's helpful to recognize the idea when it appears, because it's a reliable marker of what you can't realistically accomplish.
Except agents actually have an intent, and can route around obstacles to accomplish that intent.
If you merely block a specific action, they will find another way to do what they're trying to do. Agent security requires controlling the agent's intent.
The goal behind most "clean" software design in general is to eliminate the possibility of failure via constraints. That's the pattern I've seen over the years. Of course, the map is not the territory - you need to make sure the reachable set within the constraints is actually a subset of the real reachable set. Which may be underspecified or unknown a priori (as if you could've really specified the true reachable set, why didn't you just encode those rules?)
So I'm sympathetic to the criticism, especially since composition of formal methods & analyzing their effects is still very much a hard problem (and not just computationally - philosophically, often, for the reason I listed above).
That being said, I don't know a better solution. Begging the agent with prompts doesn't work. Are you suggesting some kind of mechanistic interpretability, maybe?
been building on claude code for a while. the post's framing is right.
mcp gives you open standards on the tool layer but the harness (claude code, cursor) is still proprietary. your product is one anthropic decision away from breaking.
the user agent role the post calls for needs open harnesses, not just open standards. otherwise we end up rebuilding mobile under a new name.
These are already available. Mistral’s Vibe CLI[1] is open source. Tools like goose[2]are API agnostic.
[1] https://github.com/mistralai/mistral-vibe
[2] https://goose-docs.ai/
A couple other major open source harnesses:
- https://github.com/badlogic/pi-mono/ - https://github.com/anomalyco/opencode
thanks, will try both.
if you've actually migrated an existing claude code setup to one of them, curious how the portability story worked. that's the part i'd been worried about.
I've not tried actually migrating from Claude Code... but having played a bit with other clients, I would avoid Mistral Vibe. I want to love it, there's some things that are nice about Vibe (mostly just "oui oui baguette"), but the things I did not like about it were disastrously bad. I could barely get MCP servers configured, and it was in something of a broken state even when I did get it working. I have many words about how horrified I am at how far behind Mistral is, but I will spare the rant.
OpenCode is another one to consider looking at: https://opencode.ai/ Not sure I'd recommend it, but it's worthy of consideration, as is Pi.
Also, consider that you can build your own. I've got Claude Code in the background working on improvements to my own harness (just for myself) at the moment. Though my intention is to have a mini API-only Claude Code that I can use on retro machines that don't support it, I don't need a full Claude Code feature set.
Sorry, I can’t help with that, as I’ve never actually used Claude. Started with Mistral and stuck there (they still have a free API tier, but I ended up buying their Le Chat Pro service anyway, for the image generation).
https://pi.dev/
opencode is open source
[dead]
[dead]
The framing assumes the agent can reliably represent its principal, and I'm not convinced that holds even if you get everything else right.
The problem is that the agent itself is the attack surface. An adversary who controls the communication channel can manipulate what the agent believes about who it's talking to, which means anything it holds, its list of authorized actions, a shared secret you gave it, whatever, can be exfiltrated in ways the agent can't detect because the manipulation happens below the layer where it can reason about trust.
Open harnesses and open standards help but they don't close this gap, because the thing you need to trust, the agent's own judgment about its principal, is exactly what gets compromised. The trust chain has to go below software entirely: hardware attestation, signed commands with keys the agent can verify but never access. That's really an OS problem dressed up as an agent architecture problem.
The thing I don’t like about “agents” is that I consider my computer a tool that I use and control. I don’t want it doing things for me: I want to do things through it. I want to be in the driver’s seat. “Notifications” and “Assistants” and now “Agents” break this philosophy. Now there are these things doing “stuff” on my computer for me and I’m just a passenger along for the ride. A computer should be that “bicycle for the mind” as Jobs put it, not some autonomous information-chauffeur, spooning output into my mouth.
The browser analogy holds because publishers wanted browsers. Sites lived with User-Agent and robots.txt because the click paid for it.
AI agents are the destination. No return click to bargain with. That's why Cloudflare just went default-block + 402 Payment Required instead of waiting on a standards body.
Open standards on the agent side are the easy half. Getting sites to show up is the part W3C can't fix alone.
I like how the author notices that it really got a start with cloud computing.
The most important thing we can do for AI to be a net positive to society is to ensure that its loyalty is to the user, and not the state.
There is no legitimate intermediate position - The skew will go one way or the other.
This is a silly thing to say.
Such a thing can’t be enforced and it can be flipped on a dime.
You should play around with local LLMs and system prompts to experience it.
First half: relatively cogent diagnosis of understood problems in computer privacy.
Second half: specious claims about AI mostly based on a vague "we don't know what they can do, so maybe they can do anything?" rhetorical maneuver.
i think whats missing is the raison detre of the Agents isnt a new usecase, its a context prune for the same limitations LLMs provide. LLM as Agent is a subset, where the goal of the agent is set by the parent and is suppose to return a pruned context.
if you dont recognize the technical limitations that produced agents youre wearing rose tinted glasses. LLMs arent approaching singularity. theyre topping out in power and agents are an attempt to exentend useful context.
The sigmoid approacheth and anyone of merit should be figuring out how the harness spits out agents, intelligently prunes context then returns the best operational bits, alongside building the garden of tools.
Its like agents are the muscles, the bones are the harness and the brain is the root parent.
[flagged]