I don't use VSCode or Copilot, so I'm hoping someone can answer these questions for me
- chmod: does Copilot run as the user? Who's file permissions does it respect?
- Can Copilot get root access?
- Can autoApprove be enabled via the standard interface? Making it possible to batch approve code changes along with this setting change?[0]
- Can it read settings from multiple files? (e.g. `.vscode/settings.json` and `../.vscode/settings.json`)
- How is the context being read? Is this in memory? File? Both?
- What happens when you edit the context? Are those changes seen in some log?
Honestly, I can't see how this problem becomes realistically solvable without hitting AGI (or pretty damn close to it). Fundamentally we have to be able to trust the thing that is writing the code and making the edits. We generally trust people because we pay, provide job security, and create a mutually beneficial system where malicious behavior is disincentivized. But a LLM doesn't really have the concept of maliciousness. Sure, we can pressure it to act certain ways but that also limits the capabilities of those tools. Can't get it to act "maliciously"? Then how is it going to properly do security testing? Now we got multiple versions of Copilot? Great, just get them to work together and you're back to where we were.
So I think the author is completely right that this gets much harrier when we let the LLMs do more and get multi-agent systems. What's the acceptable risk level? What are we willing to pay for that? It's easy to say "I'm just working on some dumb app" but honestly if it is popular enough why would this not be a target to create trojans? It's feasible for malicious people to sneak in malicious code, even when everyone is reviewing and acting diligently, but we place strong incentive structures around that to prevent this from happening. But I'm unconvinced we can do that with LLMs. And if we're being honest, it seems like letting LLMs do more erodes the incentive structure for the humans, so just makes it possible to be fighting two fronts...
So is it worth the cost? What are our limits?
[0] I'm thinking you turn it on, deploy your attack, turn it off, and the user then sees approval like they were expecting. Maybe a little longer or extra text but are they really watching the stream of text across the screen and watching every line? Seems easy to sneak in. I'm sure this can advance to be done silently or encoded in a way to make it look normal. Just have it take a temporary personality.
>I can't see how this problem becomes realistically solvable without hitting AGI
How would AGI solve this? The most common definition of AGI is "as good as average humans on all human tasks" - but in case of ITsec, that's a very low bar. We'd simply see prompt injections get more and more similar to social engineering as we approach AGI. Even if you replace "average" with "the best" it would still fall short, because human thought is not perfect. You'd really need some sort of closely aligned ASI that transcends human thought altogether. And I'm not sure if those properties aren't mutually exclusive.
That's a pretty recent definition, one developed out of marketing since it removes the need to further refine and allows it to be naïvely measured.
So I'll refine: sentient. I'll refine more: the ability to interpret the underlying intent of ill-defined goals, the ability to self generate goals, refine, reiterate, resolve and hold conflicting goals and context together, possess a theory of mind, possess triadic awareness. And I'm certain my definition is incomplete.
What I mean by AGI is the older definition: the general intelligence possessed by humans and other intelligent creatures. In context I mean much closer to a human than a cat.
It's actually one of the oldest definitions. I recommend you look up the works of H. A. Simon. This idea is quite ancient to people who are working AI research.
Anyhow, your more vague definition is still pretty much in line with my assumptions above in terms of the applicability to this issue. I.e. an AGI by your standard also will not bring a solution to this.
Maybe there's a tooling opportunity. Build some sort of local firewall that sits in front of agent calls to audit them, or at least log and track them.
I noticed that too, when working on a frontend project with hot code reloading it would immediately reflect the change even if it was still requiring review in the editor. It's convenient but also an obvious flaw that immediately turns any prompt injection into a RCE. It diligently asking me for confirmation still on every other kind of interaction feels like a dangerous false sense of security.
Why isn't this surprising? One of the early lessons, most web devs receive that functions like eval are dangerous because they allow arbitrary code execution. If you disregarding the huge financial incentives pushing for programming agents to run partially unsupervised on a dev machine on a limited selection of files, isn't it clear that something like this would happen eventually and will likely be a recurring issue?
> After reporting the vulnerability on June 29, 2025 Microsoft confirmed the repro and asked a few follow up questions. A few weeks later MSRC pointed out that it is an issue they were already tracking, and that it will be patched by August. With the August Patch Tuesday release this is now fixed.
With multiple AI agents simultaneously creating and editing multiple files, many devs won't be able to pick up malicious changes, even if they look at diffs. (And there are often pressures at work to cut corners.)
So far, I have only picked up agents overwriting files with instructions for them or creating instructions telling themselves to ignore some instructions in other files. (And pure laziness like disabling certain tests.) These are pretty obvious, could be prevented by changing file permissions (to a certain extent) and I use those more dangerously autonomous AI approaches for personal projects only. Would I pick up malicious changes if they were spread across many files, more sophisticated, and it was during crunch time? I don't know.
If there is some software that scans edits for AI-specific issues, doesn't live in VSCode, and isn't susceptible to simple prompt injection, I would happily give it a try.
I don't use VSCode or Copilot, so I'm hoping someone can answer these questions for me
Honestly, I can't see how this problem becomes realistically solvable without hitting AGI (or pretty damn close to it). Fundamentally we have to be able to trust the thing that is writing the code and making the edits. We generally trust people because we pay, provide job security, and create a mutually beneficial system where malicious behavior is disincentivized. But a LLM doesn't really have the concept of maliciousness. Sure, we can pressure it to act certain ways but that also limits the capabilities of those tools. Can't get it to act "maliciously"? Then how is it going to properly do security testing? Now we got multiple versions of Copilot? Great, just get them to work together and you're back to where we were.So I think the author is completely right that this gets much harrier when we let the LLMs do more and get multi-agent systems. What's the acceptable risk level? What are we willing to pay for that? It's easy to say "I'm just working on some dumb app" but honestly if it is popular enough why would this not be a target to create trojans? It's feasible for malicious people to sneak in malicious code, even when everyone is reviewing and acting diligently, but we place strong incentive structures around that to prevent this from happening. But I'm unconvinced we can do that with LLMs. And if we're being honest, it seems like letting LLMs do more erodes the incentive structure for the humans, so just makes it possible to be fighting two fronts...
So is it worth the cost? What are our limits?
[0] I'm thinking you turn it on, deploy your attack, turn it off, and the user then sees approval like they were expecting. Maybe a little longer or extra text but are they really watching the stream of text across the screen and watching every line? Seems easy to sneak in. I'm sure this can advance to be done silently or encoded in a way to make it look normal. Just have it take a temporary personality.
>I can't see how this problem becomes realistically solvable without hitting AGI
How would AGI solve this? The most common definition of AGI is "as good as average humans on all human tasks" - but in case of ITsec, that's a very low bar. We'd simply see prompt injections get more and more similar to social engineering as we approach AGI. Even if you replace "average" with "the best" it would still fall short, because human thought is not perfect. You'd really need some sort of closely aligned ASI that transcends human thought altogether. And I'm not sure if those properties aren't mutually exclusive.
That's a pretty recent definition, one developed out of marketing since it removes the need to further refine and allows it to be naïvely measured.
So I'll refine: sentient. I'll refine more: the ability to interpret the underlying intent of ill-defined goals, the ability to self generate goals, refine, reiterate, resolve and hold conflicting goals and context together, possess a theory of mind, possess triadic awareness. And I'm certain my definition is incomplete.
What I mean by AGI is the older definition: the general intelligence possessed by humans and other intelligent creatures. In context I mean much closer to a human than a cat.
>That's a pretty recent definition
It's actually one of the oldest definitions. I recommend you look up the works of H. A. Simon. This idea is quite ancient to people who are working AI research.
Anyhow, your more vague definition is still pretty much in line with my assumptions above in terms of the applicability to this issue. I.e. an AGI by your standard also will not bring a solution to this.
Maybe there's a tooling opportunity. Build some sort of local firewall that sits in front of agent calls to audit them, or at least log and track them.
/? llm firewall https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
I believe snyk offers something like this.
I noticed that too, when working on a frontend project with hot code reloading it would immediately reflect the change even if it was still requiring review in the editor. It's convenient but also an obvious flaw that immediately turns any prompt injection into a RCE. It diligently asking me for confirmation still on every other kind of interaction feels like a dangerous false sense of security.
Why isn't this surprising? One of the early lessons, most web devs receive that functions like eval are dangerous because they allow arbitrary code execution. If you disregarding the huge financial incentives pushing for programming agents to run partially unsupervised on a dev machine on a limited selection of files, isn't it clear that something like this would happen eventually and will likely be a recurring issue?
This is good for AI
Why submitting this again after 2 months OP?
As mentioned in the article and in previous discussions:
> With the August Patch Tuesday release this is now fixed.
I don't want to speak for OP, but isn't the idea behind responsible disclosure to give developers time to patch an exploit before publicizing it?
From the article:
> After reporting the vulnerability on June 29, 2025 Microsoft confirmed the repro and asked a few follow up questions. A few weeks later MSRC pointed out that it is an issue they were already tracking, and that it will be patched by August. With the August Patch Tuesday release this is now fixed.
>When looking at VS Code and GitHub Copilot Agent Mode I noticed a strange behavior…
Looks like only applicable to Microsoft VS "Editor". Emacs and vim users, no worry it seems.
Is there some kind of an external "AI wrangler?"
With multiple AI agents simultaneously creating and editing multiple files, many devs won't be able to pick up malicious changes, even if they look at diffs. (And there are often pressures at work to cut corners.)
So far, I have only picked up agents overwriting files with instructions for them or creating instructions telling themselves to ignore some instructions in other files. (And pure laziness like disabling certain tests.) These are pretty obvious, could be prevented by changing file permissions (to a certain extent) and I use those more dangerously autonomous AI approaches for personal projects only. Would I pick up malicious changes if they were spread across many files, more sophisticated, and it was during crunch time? I don't know.
If there is some software that scans edits for AI-specific issues, doesn't live in VSCode, and isn't susceptible to simple prompt injection, I would happily give it a try.
Or we could just not have a bunch of unpredictable LLM bots running around our systems with read/write permissions...
That's crazy talk
Great point. It's actually possible for one agent to "help" another agent to run arbitrary code and vice versa.
I call it "Cross-Agent Privilege Escalation" and described in detail how such an attack might look like with Claude Code and GitHub Copilot (https://embracethered.com/blog/posts/2025/cross-agent-privil...).
Agents that can modify their own or other agents config and security settings is something to watch out for. It's becoming a common design weakness.
As more agents operate in same environment and on same data structures we will probably see more "accidents" but also possible exploits.