I was curious to understand how Copilot implements its harness, and also how I was exhausting my quota so quickly. End up going down a rabbit hole of intercepting its network traffic with mitmproxy.
A few interesting things I found along the way:
- watched model/capability discovery and routing happen in real time
- looked at what gets injected into context and sent with ghost completions
- found that recent edits can pull in context from files other than the one you're currently editing (including infamous .env)
- found the SQLite session store behind Chronicle, including previous prompts/responses
- watched the model query that history through tool calls
I then went through the VS Code source to reconcile some of what I was seeing on the wire with the actual implementation.
Overall some interesting lessons around how their harness is implemented.
How do you actually cleanly solve that .env issue?
Anything cross platform and coding agent agnostic?
I suppose that .env file should be removed, but then things aren’t easy: no native multiplatform secret manager, or the std lib of the language doesn’t offer an API over the native secret store, etc.
Or a "secret injection proxy" for some cases could work I guess.
Infisical, or Bitwarden Secret Manager? Those two look like perfectly reasonable if the llm is just careless (but still, nothing prevents the LLM from intentionally cat'ing /proc/self/environ or from running /usr/bin/env or set or similar)
https://varlock.dev (free, open source) can pull secrets from many places, and has a credential broker (proxy) to inject placeholders, then replace with real secrets at the network boundary. There are a few other tools like this, but ours seems to be the most flexible so far.
One thing I found that I thought was a fun addition, is using eBPF made this even easier. No need to fight with anyone that is using certificate pinning, mTLS or anything else, you just get the raw plaintext data straight of the wire (right before encryption and right after decryption) and works nicely for most of the agents and IDE's.
That will in practice give you everything from telemetry to prompts, and its funny to see just how much some of them collect/run that is not at all related to your own ask..
A handy alternative when certain applications tend to make it harder to apply a MiTM proxy and you can dump it straight into your own scripts/programs to filter out and store it in whichever format you want for more analysis.
They don't offload TLS to the kernel, do they? Most apps do it in userspace linked against openssl afaik.
Do you patch that lib? If ebpf "just" operates at network/packet level, I don't see how it can do more than Mitmproxy in regard to avoid DH-PFS/Pinning
It’s done via uprobes which allows ebpf to attach to users processes or libraries, basically like ld_preload but built into the kernel. First link that explains it https://blog.quarkslab.com/defeating-ebpf-uprobe-monitoring.... but there are many more links about it and it is a pretty useful tool for debugging in prod environments. (Also first link: https://blog.px.dev/ebpf-function-tracing/) this also the intended use case.
px.dev is btw a cncf project which also helps for ebpf debugging in k8s (comes from new relic)
Exactly this :)
You can either attach the uprobes directly to the systems OpenSSL to capture anything using it (curl etc), though most of these LLM clients and similar uses their own static compiled BoringSSL, but its very straight forward to find, it does require some knowledge to determine the right place where you attach the uprobe, but after that it can easily be scripted to automatically find the right offsets (as these are usually stripped binaries)
eBPF started as a network tool, but (according to the creators of it) targeting the network was just a ploy to get a foot in the door so they could start expanding it to other things. They didn’t think the idea would be accepted if they tried to do it all at once. So now it works on many parts of the system, not just the network.
Surprised that works, I thought TLS was done entirely in process space.
I think I found it, My first thought was some sort of builtin ssl library backdoor, but it looks like you do some ld.preload shenanigans to inject a eBPF monitor. I am not sure exactly what the BPF brings to the table here. A convenient interface to intercept the accept() syscall?
It is similar but no real shenanigans but rather a common functionality called uprobes. For most of these LLM clients which uses a static compiled OpenSSL or BoringSSL the functions you attach to is also the same overall, making it easily reusable when you know what to do. There has been plenty of srticles over the time on it, quite a fun thing to play around with. I made it work already with most of them for research purposes.
Nice deep dive, I always wondered how copilot worked compared to similar tools. I'm shocked at the lack of of a rule for env files, I at least thought with a tool more integrated with github as a whole that would be a default but alas.
while this approach gives much more insights on the various requests made, if you are mainly interested in the underlying harness and its moving parts, there is already an in-built feature.
to use it, open the meatballs menu (...) of your current copilot conversation and click "show agent debug logs". it opens a tab showing all the various tool calls and prompts being sent out behind the scenes and how model selection is happening (if auto). it also gives insights on token consumption as well. moreover, vscode has been quietly shipping updates and recently you can connect your own otel service to get this information in a way you can put to use yourself.
while copilot has turned me off post their pricing changes, they have been doing tons at their own pace. highly recommend going through this feature if you find the op interesting.
I wish copilot was better at coding Java. It's like using ChatGpt 5.1, even with Fable 5 or Opus 5 as models.
The other issue with copilot is how episodic memory works. Copilot writes memories after a task is completed, which means a lot of context is lost from the intermediate exploration, success/failure steps (turns), for what? Codex's multithreaded model adds the turn outputs to episodic memory (both agents submit their episodic data to ... themselves for summary) which gives better insight when working on multi-step problems.
Disagree with the conclusion, even without carefully curated context every high end LLM perform just as well, maybe with an extra detour. In contrast if even one of the learnings is not up to date or doesn't apply to the current situation you find yourself with a long detour or even a failure.
Interesting to see how big companies make compromises with security for innovation and i feel that it's comprehensible and better that doing nothing.
But i guess it also show how we can see governance problems as real opportunity for involved peoples to build good systems with an agent native perspective.
Browser version eats about as much ram sadly. It’s not only about the Chrome dlls but also about the memory leaks, needlessly loaded resources etc etc.
It's possible, but not easy. You need to export the internal state of your TLS stack to Wireshark in order for it to decrypt the traffic. This can be done via LD_PRELOAD type shenanigans but it's difficult and fragile.
I was curious to understand how Copilot implements its harness, and also how I was exhausting my quota so quickly. End up going down a rabbit hole of intercepting its network traffic with mitmproxy.
A few interesting things I found along the way:
- watched model/capability discovery and routing happen in real time - looked at what gets injected into context and sent with ghost completions - found that recent edits can pull in context from files other than the one you're currently editing (including infamous .env) - found the SQLite session store behind Chronicle, including previous prompts/responses - watched the model query that history through tool calls
I then went through the VS Code source to reconcile some of what I was seeing on the wire with the actual implementation.
Overall some interesting lessons around how their harness is implemented.
How do you actually cleanly solve that .env issue?
Anything cross platform and coding agent agnostic?
I suppose that .env file should be removed, but then things aren’t easy: no native multiplatform secret manager, or the std lib of the language doesn’t offer an API over the native secret store, etc.
Or a "secret injection proxy" for some cases could work I guess.
Infisical, or Bitwarden Secret Manager? Those two look like perfectly reasonable if the llm is just careless (but still, nothing prevents the LLM from intentionally cat'ing /proc/self/environ or from running /usr/bin/env or set or similar)
https://varlock.dev (free, open source) can pull secrets from many places, and has a credential broker (proxy) to inject placeholders, then replace with real secrets at the network boundary. There are a few other tools like this, but ours seems to be the most flexible so far.
Do you feel that something like https://secretspec.dev/ addresses these points?
Data retention clauses?
I dont see how you can ever really trust an LLM anyway to follow instructions.
[flagged]
One thing I found that I thought was a fun addition, is using eBPF made this even easier. No need to fight with anyone that is using certificate pinning, mTLS or anything else, you just get the raw plaintext data straight of the wire (right before encryption and right after decryption) and works nicely for most of the agents and IDE's.
That will in practice give you everything from telemetry to prompts, and its funny to see just how much some of them collect/run that is not at all related to your own ask..
A handy alternative when certain applications tend to make it harder to apply a MiTM proxy and you can dump it straight into your own scripts/programs to filter out and store it in whichever format you want for more analysis.
Out of curiosity: How?
They don't offload TLS to the kernel, do they? Most apps do it in userspace linked against openssl afaik.
Do you patch that lib? If ebpf "just" operates at network/packet level, I don't see how it can do more than Mitmproxy in regard to avoid DH-PFS/Pinning
It’s done via uprobes which allows ebpf to attach to users processes or libraries, basically like ld_preload but built into the kernel. First link that explains it https://blog.quarkslab.com/defeating-ebpf-uprobe-monitoring.... but there are many more links about it and it is a pretty useful tool for debugging in prod environments. (Also first link: https://blog.px.dev/ebpf-function-tracing/) this also the intended use case. px.dev is btw a cncf project which also helps for ebpf debugging in k8s (comes from new relic)
Exactly this :) You can either attach the uprobes directly to the systems OpenSSL to capture anything using it (curl etc), though most of these LLM clients and similar uses their own static compiled BoringSSL, but its very straight forward to find, it does require some knowledge to determine the right place where you attach the uprobe, but after that it can easily be scripted to automatically find the right offsets (as these are usually stripped binaries)
eBPF started as a network tool, but (according to the creators of it) targeting the network was just a ploy to get a foot in the door so they could start expanding it to other things. They didn’t think the idea would be accepted if they tried to do it all at once. So now it works on many parts of the system, not just the network.
Surprised that works, I thought TLS was done entirely in process space.
I think I found it, My first thought was some sort of builtin ssl library backdoor, but it looks like you do some ld.preload shenanigans to inject a eBPF monitor. I am not sure exactly what the BPF brings to the table here. A convenient interface to intercept the accept() syscall?
It is similar but no real shenanigans but rather a common functionality called uprobes. For most of these LLM clients which uses a static compiled OpenSSL or BoringSSL the functions you attach to is also the same overall, making it easily reusable when you know what to do. There has been plenty of srticles over the time on it, quite a fun thing to play around with. I made it work already with most of them for research purposes.
[dead]
Minor factual correction: The Codex client is open source. https://github.com/openai/codex
thanks! corrected in the article
Nice deep dive, I always wondered how copilot worked compared to similar tools. I'm shocked at the lack of of a rule for env files, I at least thought with a tool more integrated with github as a whole that would be a default but alas.
while this approach gives much more insights on the various requests made, if you are mainly interested in the underlying harness and its moving parts, there is already an in-built feature.
to use it, open the meatballs menu (...) of your current copilot conversation and click "show agent debug logs". it opens a tab showing all the various tool calls and prompts being sent out behind the scenes and how model selection is happening (if auto). it also gives insights on token consumption as well. moreover, vscode has been quietly shipping updates and recently you can connect your own otel service to get this information in a way you can put to use yourself.
while copilot has turned me off post their pricing changes, they have been doing tons at their own pace. highly recommend going through this feature if you find the op interesting.
Nice one! Really shows why we should run those in sandboxes without env access. I like the proxy swap approach
I wish copilot was better at coding Java. It's like using ChatGpt 5.1, even with Fable 5 or Opus 5 as models.
The other issue with copilot is how episodic memory works. Copilot writes memories after a task is completed, which means a lot of context is lost from the intermediate exploration, success/failure steps (turns), for what? Codex's multithreaded model adds the turn outputs to episodic memory (both agents submit their episodic data to ... themselves for summary) which gives better insight when working on multi-step problems.
I have found beads works pretty well for this
I will try it. npm install -g @beads/bd
Disagree with the conclusion, even without carefully curated context every high end LLM perform just as well, maybe with an extra detour. In contrast if even one of the learnings is not up to date or doesn't apply to the current situation you find yourself with a long detour or even a failure.
Interesting to see how big companies make compromises with security for innovation and i feel that it's comprehensible and better that doing nothing. But i guess it also show how we can see governance problems as real opportunity for involved peoples to build good systems with an agent native perspective.
When I did this I just used a custom token in visual studio and then looked at the logs in the provider's web UI.
I don't think they care very much about this. The encrypted blobs for reasoning models is a different matter.
We stopped trying to keep the key away from the model and made it cheap instead: per-run, spend-capped, deleted at teardown. Assume it reads the file.
"Apps users love, made with Electron". Seriously?
They use those apps either because of network effects or because there are few alternatives.
or completely deleting the app and use the browser version for desperation
Browser version eats about as much ram sadly. It’s not only about the Chrome dlls but also about the memory leaks, needlessly loaded resources etc etc.
I believe this can be done without a MitM by using Wireshark or adjacent tool
It's possible, but not easy. You need to export the internal state of your TLS stack to Wireshark in order for it to decrypt the traffic. This can be done via LD_PRELOAD type shenanigans but it's difficult and fragile.
MITM proxy is much easier to get working.
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]