Looks great! Reading through the docs it seems the subtrace process sends all data to your server. I'm reluctant to do that on a production environment, where API keys and personal data are being handled.
Is there any way to run it completely self hosted? If not, are there plans? And how will you monitize self hosted options (if it's possible)?
Pixie [1] is a similar project and offers the self hosted model you are looking for.
We also support 11 application protocols [2] with TLS handshake tracing and MQTT support coming soon (encrypted traffic tracing has been supported for a long time).
From a dictionary: The meaning of DISCLAIMER is a denial or disavowal of legal claim : relinquishment of or formal refusal to accept an interest or estate.
re self-hosting: yep! Use the -devtools flag to get a stripped down version of Subtrace running entirely locally:
subtrace run -devtools=/subtrace -- python3 -m http.server
This starts a Python server on localhost:8000 but with Subtrace. Everything except /subtrace is forwarded to the Python server like usual, but if you go to http://localhost:8000/subtrace you should see the Chrome DevTools network tab running in the browser like a regular app. Any request you send to localhost:8000 + all outgoing requests made from inside the Python app will automatically appear in that dashboard!
That would work for a single instance, but when running multiple (e.g., you are horizontally scaled) it would not be ideal.
Is it possible to mimic "subtrace.dev"? There's the 'SUBTRACE_ENDPOINT' environment variable which can be used to set the target endpoint, but is the server side open source too? And does the license grant permission for self hosting the full stack?
re the Wireshark analogy: the reason I used that was because: (1) Subtrace operates at roughly the same level in the operating system stack, (2) has similar capabilities, (3) has an overlap in use-cases, and (4) has been the most effective at communicating what Subtrace is in my experience so far. I can see why the analogy is not a perfect 1:1 mapping (obligatory xkcd: https://xkcd.com/624), but naming things is hard and taglines are just names in idea space :)
My first impression with "Wireshark for Docker containers" is "... Well, Wireshark already works with Docker containers, so if your goal is to be Wireshark for Docker containers, you'll just fail by definition. Can't beat Wireshark at being Wireshark."
I'm saying this just FYI. I haven't actually looked at what your product does, but if it were to matter to me, it'd be based on what it can offer that Wireshark can't, rather than how similar it is to Wireshark.
I'm struggling to understand why I wouldn't just use wireshark and specify the interface as well. Reading docs shows you need to invoke your container with this as a wrapper which to me is quite a bit scarier than attaching to an interface.
Can I download a pcap file? Being able to view and debug the application protocol is one thing, but I can use Wireshark to view and debug TCP itself.
Reading the documentation makes it sound like this sits one or two levels above where Wireshark usually operates, which is why I think the analogy is tough.
Wireshark is a weird choice in my opinion because you know, why would i use Subtrace if i can use Wireshark?
and People who don't know how to use Wireshark want probably a better motivation to use Subtrace than 'its like wireshark' because there is a reason why they don't use it.
Since we operate at the TCP level, we can actually handle pretty much any protocol. I have an implementation of a postgres handler in my git stash that intercepts and shows the SQL queries executed + the resulting rows alongside the HTTP request that triggered it (I still need to do some robustness and correctness testing before it's ready to merge). With a handful of other protocols like MySQL, Mongo, Redis, Kafka, or even FTP lol, I think Subtrace can cover most practical dev workloads.
Btw Subtrace can already record .pcap files today since it's just a simple TCP stream proxy, but raw network packet captures are mostly only useful when you're implementing new protocols, which 99% of the people using Docker containers today aren't doing. It's also a solved problem because you can just run `apt-get install tcpdump` inside the container.
Automatic tracing for app-level protocols that is easy to setup, works everywhere, lightweight for prod, fast to search, and can show the data in a clean interface is still insanely difficult today. That's the problem Subtrace is trying to solve.
It's a pretty cool looking product. It's not wireshark, it's not close to wireshark just because it can capture some tcp pcaps, and there are more protocols relevant to container networking than a handful of TCP app-level protocols.
When I came in I was hoping to see a product that actually was for container networking, not just app data flows. Again - this is a neat tool, and probably incredibly useful for people developing way up the stack like that, but a lot of us live below the bottom of a "full-stack developer's" stack. Some features I would expect in a "wireshark for Docker containers":
* Ability to inspect DNS traffic
* Ability to trace packets the enter the conainter network stack (e.g. the packet(s) generated when the server calls send() ) into the virtual interface for the namespace and through the host machine. Ideally with multiple observation points and correlation of packets in the overlay/underlay contexts (decrypting from local keys whenever possible).
* Ability to see where a packet dies between the app and exit NIC on the host. Including things like "packets delivered to this container even though the dest IP/subnet isn't in this container"
* Similar to the previous point: ability to track packets through all the NAT steps containers introduce.
* See arp traffic on virtual interfaces.
* Ability to observe the TLS handshake and gather all the parameters of the connection.
* Packet dissection and protocol session tracing for all the tunnels.
* Bonus points if you can capture weird teleports caused by ebpf programs in the network path.
I expect this because that's how I use wireshark_+ bpftrace in container environments for the most part. I've also used it to debug while implementing protocols, but that's a less common use case of packet dissection and tracing in wireshark.
What you've built is cool, and I can see it expanding in a lot of directions very, very, usefully. I just really dislike something calling itself a wireshark while not really helping with networking (and in fact - the networking has to work reasonably well for this tool to be effective).
That's totally fair, I can see why Wireshark wouldn't be the most accurate description to someone working on those kinds of problems. And fwiw, I wish my problems (before Subtrace) were cool enough to need to whip out Wireshark for packet-level inspection :)
Yes, but we've managed to do it automatically without any library/language specific hooks! It's probably one of my favourite things in Subtrace :)
We generate an ephemeral TLS root CA certificate and inject it into the system store. The generated certificate is entirely in-memory and never leaves the machine. To make this work without root privileges, we intercept the open(2) syscall to see if it's /etc/ssl/certs/ca-certificates.crt (or equivalent). If so, we append the ephemeral root CA to the list of actual CA certificates; if not, we let the kernel handle the file open like usual. This way, none of the other programs are affected, so only the program you start with `subtrace run` sees and trusts the ephemeral root CA.
After we get the program to trust the ephemeral root CA, we can proxy outgoing TLS connections through Subtrace transparently but also read the cleartext bytes.
It won't work with programs that defensively validate the cert chain but those are rare.
It won't work with programs that embed their own root cert store, which is also rare but I would guess less rare than the previous one. The usual reason to do this is to minimize OS deps, and in the case of Docker containers to save on container image size by only including the roots you care about.
But yes for the vast majority of programs it should work fine.
Yep, certificate pinning is the one scenario Subtrace can't handle in my experience, but thankfully, it's fairly rare like you said. And IMO there is no general solution to the problem [1], but it's one of those very interesting problems to daydream thinking about when you're stuck in traffic or whatever :)
We still try our best by handling as much of the long tail of environments with some library/framework specific workaround (e.g. Deno bundles all TLS certs in its binary so we set the DENO_CERT env var when applicable).
kubeshark [0] is using ebpf to catch calls to openssl/go's tls lib and thus no need to juggle certs. Has pros and cons compared to your method, but an interesting comparison.
Is this a different method from the httptap [1] that was on hackernews a few weeks ago? Somebody in that post seemed to say that it also generates CA certificates on the fly.
httptap is really cool! Their technique is different (they do a filesystem mount instead of intercepting syscalls like Subtrace does) but both tools effectively reach the same goal using different routes.
Up front, this is not my area of expertise. You would still not want to run this on the same server as your containers since someone could inject their own cert? I think it allows someone to decrypt traffic that isn't just proxied to Subtrace?
Edit: I've reviewed the docs and it looks like you do run it on the same server. For clarity, I've used Sentry before.
Subtrace proxies the program's connection using a regular TLS connection to the upstream server. For example, if you do `subtrace run -- curl https://example.com`, curl thinks it's talking to example.com over TLS, but it's really talking to Subtrace locally. Since we injected the ephemeral root CA into the system store, curl will trust the valid TLS certificate that Subtrace presents for example.com. From within the same server, Subtrace will handle the actual TLS connection to upstream example.com. That upstream connection is undecipherable to outsiders.
Everything is exactly as secure as before Subtrace. In other words, using Subtrace doesn't make the NSA's job any easier ;)
This is a great hack. It won’t work 100% of the time but it’ll be close, and damn it’s just so clean. Proxies and intercepts all the way down… I love this.
You can use Subtrace entirely locally without talking to subtrace.dev using the -devtools flag. For example:
subtrace run -devtools=/subtrace -- python3 -m http.server
Then go to http://localhost:8000/subtrace and you'll see the Chrome DevTools interface running in your browser like a regular app. Any request sent to http://localhost:8000 should appear there in realtime. Note that the -devtools flag is kinda best effort because Subtrace is intended to be used in production.
We have fully on-prem installations for enterprises, of course (e.g. if you're in a highly regulated industry like healthcare).
My most painful debugging scenarios with Docker networking (for me) has always been dealing with non-TCP traffic. But still, this seems useful. One thing I don't understand is why this requires an account token? Does this require a network connection to subtrace? It seems like this should all be running locally, and these kinds of connection details are _exactly_ the kind I would not want to leave the host, let alone go to a third party.
One of my litmus tests for software is "can I use it on a flight?", so you'll be pleased to know Subtrace works great locally without ever talking to subtrace.dev!
subtrace run -devtools=/subtrace -- python3 -m http.server
Run the above command in a Linux machine, go to http://localhost:8000/subtrace and send some requests to localhost:8000 to see a stripped down version of the Subtrace dashboard working fully locally.
Congrats on the seccomp-based interception, that's a really neat way to solve this problem! We did some BPF_PROG_TYPE_CGROUP_SOCK eBPF shenanigans in mitmproxy for redirection, but that doesn't work with containers at all. Cool to see that intercepting all relevant syscalls works that well.
Have not played around with it, but, curious, how does debugging on production work for a specific request/session? Can I filter by some sort if request trace id or something?
You can tag each request with arbitrary key-value maps and search over these later. For example, if you add a `x-subtrace-tags: user=foo, project=bar` header on the response, you can apply a `tags.user == "foo"` filter in the dashboard to see all requests across your entire infra from that user and only that user. Each request is pre-populated by default with tags like hostname, pod name, AWS/GCP location, etc.
It's like Honeycomb's wide events but even better because: (1) you can see whole request including the payload alongside the event fields, and (2) it's fully automatic and requires no code changes out of the box (you can incrementally add these tags when you find a need for each one instead of the huge upfront cost from instrumenting the hell out of your codebase).
So "tcpdump as a service"? Why wouldn't I just generate my own pcap and stick it into wireshark or whatever I like for looking at packet captures? I'm having trouble seeing the value prop here.
One of things we're thinking about is automatic method/function call tracing. Something like attaching the entire stack trace of calls done to handle the API request. Ideally using the same UI so that you can see the headers/payload that was sent and the function-level stack trace right next to each other. None of the OpenTelemetry verbosity, all of the observability!
More than 0.01 ms, less than 0.10 ms. Noticeable when you're a high frequency trading firm, probably not otherwise :)
In the hot path, Subtrace is just a dumb proxy that copies bytes. All of the processing + indexing happens offline in a different machine (in Clickhouse).
We also have an on-prem version of Subtrace for enterprises. It runs in their own AWS account without ever talking to subtrace.dev so that companies in regulated industries like healthcare can use Subtrace.
> Stratoshark captures and analyzes system calls and logs using libsinsp and libscap, and can share capture files with the Sysdig command line tool and Falco
Not in the very short term, but we've talked about it internally and we might do it in the future!
If you'd like to use Subtrace on Windows, it would be super helpful to understand your use-case deeply so that we build the right things in the right order. Please reach out to me at adtac@subtrace.dev, I'd love to chat!
You can use mitmproxy and mitmweb to achieve the same. It is in Docker hub and you can pass environment variables to your other containers to make it work.
The TLS certificate setup is more tricky but that is always going to be a pain.
Burp Proxy is another great tool that is even more powerful but harder to set up.
Pixie (https://px.dev) can be installed in under 5 mins and gives this level of visibility across all applications. No need to change your application (wrap in `subtrace run`) to get instant visibility.
We also support 11 application protocols (https://docs.px.dev/reference/datatables/) with TLS handshake tracing and MQTT support coming soon (encrypted traffic tracing has been supported for a long time).
I'm working on an even simpler way where you can just `kubectl apply` a DaemonSet or a Helm chart to get automatic tracing for all pods in your cluster instantly without any code-level changes. If anyone is interested in beta testing this, email me at adtac@subtrace.dev, I'd love to understand your usecase!
It’d be neat to use subtrace in an ephemeral pod for debugging purposes, that just runs alongside the regular pod.
For monitoring the network traffic for the whole cluster, the CNI and/or whatever ebpf-based runtime security stuff you’re using (falco, tetragon, tracee) is usually enough, but I can definitely see the usefulness of subtract for more specific debugging purposes. If run as a DaemonSet make sure to add some pod filtering such as namespace and label selectors (but I’m sure you’ve already thought about that).
> use subtrace in an ephemeral pod for debugging purposes
That's a great suggestion. It'd be like kubectl exec-ing into a shell inside the pod, but for network activity. I think I'm going to prototype this tonight :)
> pod filtering such as namespace and label selectors
Yep, Subtrace already tags each request with a bunch of metadata about the place where it originated so that you can filter on those in the dashboard :) Things like the hostname, pod, cluster, AWS/GCP location are automatically populated, but you can also set custom tags in the config [1].
Might be an overkill to use it in this capacity but SUSE NeuVector (https://github.com/neuvector/neuvector) can build a connectivity graph between pods and allows to perform packet capture to PCAP. It works at the virtual switch layer and can capture all traffic.
They can, but for open-source startups I recommend they do a Show HN first, for the following reasons:
(1) If the Show HN succeeds, then they can save their Launch HN card to play later; if it doesn't succeed, they can go ahead and do a Launch HN soon afterwards.
(2) Show HN is more about the project/tech, especially when it's open-source. Launch HN is more about the startup/business.
(3) Nowadays many startups offer an open-source (or open core) version for free, then make money with a hosted version. For these, it's a good fit to first do a Show HN about the open-source offering, and then later do a Launch HN about the cloud product.
There needs to be a new term for this grassroots astroturf... Grasstroturf? Get the nerds' attention with free tech, down-playing the fact that it's a business, so later they can pump them for money easier. The pig doesn't want to see the knife, they just want the free trough.
FWIW, I've spent time at YC open-source events where later-stage founders advise earlier-stage ones, and this isn't the kind of advice they give. In fact they would describe that as a terrible strategy and tell startups to avoid it.
If I were to give an example of actual advice I've heard, using the terms of your comment, it might be: keep the nerds happy with free open-source software that actually works, then charge money for things that companies (especially enterprise companies) actually prefer to pay for.
The "paid hosted offering" pattern is the most common of these, since nerds might want to run their own instances but many IT departments do not.
Looks great! Reading through the docs it seems the subtrace process sends all data to your server. I'm reluctant to do that on a production environment, where API keys and personal data are being handled.
Is there any way to run it completely self hosted? If not, are there plans? And how will you monitize self hosted options (if it's possible)?
Disclaimer: I'm a maintainer of the project
Pixie [1] is a similar project and offers the self hosted model you are looking for.
We also support 11 application protocols [2] with TLS handshake tracing and MQTT support coming soon (encrypted traffic tracing has been supported for a long time).
[1] https://px.dev
[2] https://docs.px.dev/reference/datatables/
From a dictionary: The meaning of DISCLAIMER is a denial or disavowal of legal claim : relinquishment of or formal refusal to accept an interest or estate.
Perhaps you meant DISCLOSURE
re self-hosting: yep! Use the -devtools flag to get a stripped down version of Subtrace running entirely locally:
This starts a Python server on localhost:8000 but with Subtrace. Everything except /subtrace is forwarded to the Python server like usual, but if you go to http://localhost:8000/subtrace you should see the Chrome DevTools network tab running in the browser like a regular app. Any request you send to localhost:8000 + all outgoing requests made from inside the Python app will automatically appear in that dashboard!That would work for a single instance, but when running multiple (e.g., you are horizontally scaled) it would not be ideal.
Is it possible to mimic "subtrace.dev"? There's the 'SUBTRACE_ENDPOINT' environment variable which can be used to set the target endpoint, but is the server side open source too? And does the license grant permission for self hosting the full stack?
Wireshark seems a bit misleading. More like a "network inspector" if one leans towards the browser's network tab in the inspector?
But it really looks useful and I'll definitely play with it to see if I put it into my toolbox.
Thanks!
re the Wireshark analogy: the reason I used that was because: (1) Subtrace operates at roughly the same level in the operating system stack, (2) has similar capabilities, (3) has an overlap in use-cases, and (4) has been the most effective at communicating what Subtrace is in my experience so far. I can see why the analogy is not a perfect 1:1 mapping (obligatory xkcd: https://xkcd.com/624), but naming things is hard and taglines are just names in idea space :)
My first impression with "Wireshark for Docker containers" is "... Well, Wireshark already works with Docker containers, so if your goal is to be Wireshark for Docker containers, you'll just fail by definition. Can't beat Wireshark at being Wireshark."
I'm saying this just FYI. I haven't actually looked at what your product does, but if it were to matter to me, it'd be based on what it can offer that Wireshark can't, rather than how similar it is to Wireshark.
I'm struggling to understand why I wouldn't just use wireshark and specify the interface as well. Reading docs shows you need to invoke your container with this as a wrapper which to me is quite a bit scarier than attaching to an interface.
Can I download a pcap file? Being able to view and debug the application protocol is one thing, but I can use Wireshark to view and debug TCP itself.
Reading the documentation makes it sound like this sits one or two levels above where Wireshark usually operates, which is why I think the analogy is tough.
Wireshark is a weird choice in my opinion because you know, why would i use Subtrace if i can use Wireshark?
and People who don't know how to use Wireshark want probably a better motivation to use Subtrace than 'its like wireshark' because there is a reason why they don't use it.
I can already use Wireshark itself with docker containers.
IMO the real UX problem is that there’s nothing in between “Wireshark” and “DevTools.”
What would be cool is a program that renders the real DevTools UI from a .pcap file (or stdin).
There’s a lot of use cases where I can capture all the traffic with Wireshark but I just want a basic DevTools interface.
Looks like it is for http requests only? If so, wireshark is not an apt comparison.
For now, yes :)
Since we operate at the TCP level, we can actually handle pretty much any protocol. I have an implementation of a postgres handler in my git stash that intercepts and shows the SQL queries executed + the resulting rows alongside the HTTP request that triggered it (I still need to do some robustness and correctness testing before it's ready to merge). With a handful of other protocols like MySQL, Mongo, Redis, Kafka, or even FTP lol, I think Subtrace can cover most practical dev workloads.
Btw Subtrace can already record .pcap files today since it's just a simple TCP stream proxy, but raw network packet captures are mostly only useful when you're implementing new protocols, which 99% of the people using Docker containers today aren't doing. It's also a solved problem because you can just run `apt-get install tcpdump` inside the container.
Automatic tracing for app-level protocols that is easy to setup, works everywhere, lightweight for prod, fast to search, and can show the data in a clean interface is still insanely difficult today. That's the problem Subtrace is trying to solve.
It's a pretty cool looking product. It's not wireshark, it's not close to wireshark just because it can capture some tcp pcaps, and there are more protocols relevant to container networking than a handful of TCP app-level protocols.
When I came in I was hoping to see a product that actually was for container networking, not just app data flows. Again - this is a neat tool, and probably incredibly useful for people developing way up the stack like that, but a lot of us live below the bottom of a "full-stack developer's" stack. Some features I would expect in a "wireshark for Docker containers":
* Ability to inspect DNS traffic
* Ability to trace packets the enter the conainter network stack (e.g. the packet(s) generated when the server calls send() ) into the virtual interface for the namespace and through the host machine. Ideally with multiple observation points and correlation of packets in the overlay/underlay contexts (decrypting from local keys whenever possible).
* Ability to see where a packet dies between the app and exit NIC on the host. Including things like "packets delivered to this container even though the dest IP/subnet isn't in this container"
* Similar to the previous point: ability to track packets through all the NAT steps containers introduce.
* See arp traffic on virtual interfaces.
* Ability to observe the TLS handshake and gather all the parameters of the connection.
* Packet dissection and protocol session tracing for all the tunnels.
* Bonus points if you can capture weird teleports caused by ebpf programs in the network path.
I expect this because that's how I use wireshark_+ bpftrace in container environments for the most part. I've also used it to debug while implementing protocols, but that's a less common use case of packet dissection and tracing in wireshark.
What you've built is cool, and I can see it expanding in a lot of directions very, very, usefully. I just really dislike something calling itself a wireshark while not really helping with networking (and in fact - the networking has to work reasonably well for this tool to be effective).
That's totally fair, I can see why Wireshark wouldn't be the most accurate description to someone working on those kinds of problems. And fwiw, I wish my problems (before Subtrace) were cool enough to need to whip out Wireshark for packet-level inspection :)
Can it decrypt tls? Perhaps by hooking the calls to common libraries?
Yes, but we've managed to do it automatically without any library/language specific hooks! It's probably one of my favourite things in Subtrace :)
We generate an ephemeral TLS root CA certificate and inject it into the system store. The generated certificate is entirely in-memory and never leaves the machine. To make this work without root privileges, we intercept the open(2) syscall to see if it's /etc/ssl/certs/ca-certificates.crt (or equivalent). If so, we append the ephemeral root CA to the list of actual CA certificates; if not, we let the kernel handle the file open like usual. This way, none of the other programs are affected, so only the program you start with `subtrace run` sees and trusts the ephemeral root CA.
After we get the program to trust the ephemeral root CA, we can proxy outgoing TLS connections through Subtrace transparently but also read the cleartext bytes.
All of this is fully automatic, of course.
This will not work with HPKP but hopefully nothing is using that any more. ( https://en.m.wikipedia.org/wiki/HTTP_Public_Key_Pinning )
It won't work with programs that defensively validate the cert chain but those are rare.
It won't work with programs that embed their own root cert store, which is also rare but I would guess less rare than the previous one. The usual reason to do this is to minimize OS deps, and in the case of Docker containers to save on container image size by only including the roots you care about.
But yes for the vast majority of programs it should work fine.
Yep, certificate pinning is the one scenario Subtrace can't handle in my experience, but thankfully, it's fairly rare like you said. And IMO there is no general solution to the problem [1], but it's one of those very interesting problems to daydream thinking about when you're stuck in traffic or whatever :)
We still try our best by handling as much of the long tail of environments with some library/framework specific workaround (e.g. Deno bundles all TLS certs in its binary so we set the DENO_CERT env var when applicable).
[1] https://news.ycombinator.com/item?id=42923998
No general solution? :) If you came up with this hack I’m sure you can extend it…
Cert pinning has to read a public cert from memory, right? And a public cert has a well-known shape… and you have bpf and access to the memory…
in a different context, maybe: https://news.ycombinator.com/item?id=38044786 :)
kubeshark [0] is using ebpf to catch calls to openssl/go's tls lib and thus no need to juggle certs. Has pros and cons compared to your method, but an interesting comparison.
[0] https://www.kubeshark.co/
The approach you describe above is common for similar projects:
- Pixie (https://px.dev) -- which I contribute to
- Beyla (https://github.com/grafana/beyla)
- Coroot (https://github.com/coroot/coroot)
If you are interested in the details and how the strategy for this tracing has evolved, you can learn more in this blog (https://blog.px.dev/ebpf-tls-tracing-past-present-future/).
Is this a different method from the httptap [1] that was on hackernews a few weeks ago? Somebody in that post seemed to say that it also generates CA certificates on the fly.
[1] https://news.ycombinator.com/item?id=42919909
httptap is really cool! Their technique is different (they do a filesystem mount instead of intercepting syscalls like Subtrace does) but both tools effectively reach the same goal using different routes.
Up front, this is not my area of expertise. You would still not want to run this on the same server as your containers since someone could inject their own cert? I think it allows someone to decrypt traffic that isn't just proxied to Subtrace?
Edit: I've reviewed the docs and it looks like you do run it on the same server. For clarity, I've used Sentry before.
Subtrace proxies the program's connection using a regular TLS connection to the upstream server. For example, if you do `subtrace run -- curl https://example.com`, curl thinks it's talking to example.com over TLS, but it's really talking to Subtrace locally. Since we injected the ephemeral root CA into the system store, curl will trust the valid TLS certificate that Subtrace presents for example.com. From within the same server, Subtrace will handle the actual TLS connection to upstream example.com. That upstream connection is undecipherable to outsiders.
Everything is exactly as secure as before Subtrace. In other words, using Subtrace doesn't make the NSA's job any easier ;)
That's helpful! Thanks for clarifying. I'll have to check it out.
Wow. I think more network inspection tools should adopt an approach like this. Great job.
This is a great hack. It won’t work 100% of the time but it’ll be close, and damn it’s just so clean. Proxies and intercepts all the way down… I love this.
I can't decide if I'm horrified or amazed by this :)
as with anything too clever and undocumented, you will be amazed until you waste a few weeks debugging an issue caused by it.
From the video it seems the dashboard is hosted on sibtrace.dev. Is my data being sent there? That's an absolute no-go for me.
You can use Subtrace entirely locally without talking to subtrace.dev using the -devtools flag. For example:
Then go to http://localhost:8000/subtrace and you'll see the Chrome DevTools interface running in your browser like a regular app. Any request sent to http://localhost:8000 should appear there in realtime. Note that the -devtools flag is kinda best effort because Subtrace is intended to be used in production.We have fully on-prem installations for enterprises, of course (e.g. if you're in a highly regulated industry like healthcare).
This is such a cool product! I was looking through the docs and I think this could be a lot more prominent. Thanks for building in public!
My most painful debugging scenarios with Docker networking (for me) has always been dealing with non-TCP traffic. But still, this seems useful. One thing I don't understand is why this requires an account token? Does this require a network connection to subtrace? It seems like this should all be running locally, and these kinds of connection details are _exactly_ the kind I would not want to leave the host, let alone go to a third party.
One of my litmus tests for software is "can I use it on a flight?", so you'll be pleased to know Subtrace works great locally without ever talking to subtrace.dev!
Run the above command in a Linux machine, go to http://localhost:8000/subtrace and send some requests to localhost:8000 to see a stripped down version of the Subtrace dashboard working fully locally.Congrats on the seccomp-based interception, that's a really neat way to solve this problem! We did some BPF_PROG_TYPE_CGROUP_SOCK eBPF shenanigans in mitmproxy for redirection, but that doesn't work with containers at all. Cool to see that intercepting all relevant syscalls works that well.
Have not played around with it, but, curious, how does debugging on production work for a specific request/session? Can I filter by some sort if request trace id or something?
You can tag each request with arbitrary key-value maps and search over these later. For example, if you add a `x-subtrace-tags: user=foo, project=bar` header on the response, you can apply a `tags.user == "foo"` filter in the dashboard to see all requests across your entire infra from that user and only that user. Each request is pre-populated by default with tags like hostname, pod name, AWS/GCP location, etc.
It's like Honeycomb's wide events but even better because: (1) you can see whole request including the payload alongside the event fields, and (2) it's fully automatic and requires no code changes out of the box (you can incrementally add these tags when you find a need for each one instead of the huge upfront cost from instrumenting the hell out of your codebase).
Always wanted a tool like this. Will try it out next time I need to inspect traffic of a docker container.
So "tcpdump as a service"? Why wouldn't I just generate my own pcap and stick it into wireshark or whatever I like for looking at packet captures? I'm having trouble seeing the value prop here.
Very happy subtrace user here. Especially useful to possess the Server-Timing headers.
Can you share more about how you are doing this?
IIRC, Server-Timing headers are Firefox-only and they are not available via fetch or XHR.
standards ftw!
This will help those developers who don't want to navigate Wireshark's setup and just want the familiar Chrom DevTools. Very cool project!
Do you envision going beyond just network calls from bpf?
What would you like to see?
One of things we're thinking about is automatic method/function call tracing. Something like attaching the entire stack trace of calls done to handle the API request. Ideally using the same UI so that you can see the headers/payload that was sent and the function-level stack trace right next to each other. None of the OpenTelemetry verbosity, all of the observability!
Interesting! How does this handle latency? Does it introduce any noticeable delay?
More than 0.01 ms, less than 0.10 ms. Noticeable when you're a high frequency trading firm, probably not otherwise :)
In the hot path, Subtrace is just a dumb proxy that copies bytes. All of the processing + indexing happens offline in a different machine (in Clickhouse).
Cool product, I can imagine my engineers using it. Out of curiosity, how would you monitize that?
Flat pricing with unlimited requests hosted in the cloud (https://subtrace.dev/pricing).
We also have an on-prem version of Subtrace for enterprises. It runs in their own AWS account without ever talking to subtrace.dev so that companies in regulated industries like healthcare can use Subtrace.
stratoshark, the docker container part of wireshark, may be a better match for that description.
I'd probably use a postman related pitch instead. This is much closer to that and looks like a nice complement to that workflow
Stratoshark was discussed last month https://news.ycombinator.com/item?id=42793777
Thanks! Macroexpanded:
Show HN: Stratoshark, a sibling application to Wireshark - https://news.ycombinator.com/item?id=42793777 - Jan 2025 (50 comments)
Stratoshark: https://wiki.wireshark.org/Stratoshark :
> Stratoshark captures and analyzes system calls and logs using libsinsp and libscap, and can share capture files with the Sysdig command line tool and Falco
We use https://treblle.com/ at work for this in production. Very handy to see what requests are being made and by whom.
Any Windows Server support coming?
Not in the very short term, but we've talked about it internally and we might do it in the future!
If you'd like to use Subtrace on Windows, it would be super helpful to understand your use-case deeply so that we build the right things in the right order. Please reach out to me at adtac@subtrace.dev, I'd love to chat!
You can use mitmproxy and mitmweb to achieve the same. It is in Docker hub and you can pass environment variables to your other containers to make it work.
The TLS certificate setup is more tricky but that is always going to be a pain.
Burp Proxy is another great tool that is even more powerful but harder to set up.
anything similar for k8s?
Disclaimer: I'm a maintainer of the project
Pixie (https://px.dev) can be installed in under 5 mins and gives this level of visibility across all applications. No need to change your application (wrap in `subtrace run`) to get instant visibility.
We also support 11 application protocols (https://docs.px.dev/reference/datatables/) with TLS handshake tracing and MQTT support coming soon (encrypted traffic tracing has been supported for a long time).
Subtrace already works great on Kubernetes (https://docs.subtrace.dev/kubernetes)! Add a single line to your image's Dockerfile and that's it.
I'm working on an even simpler way where you can just `kubectl apply` a DaemonSet or a Helm chart to get automatic tracing for all pods in your cluster instantly without any code-level changes. If anyone is interested in beta testing this, email me at adtac@subtrace.dev, I'd love to understand your usecase!
It’d be neat to use subtrace in an ephemeral pod for debugging purposes, that just runs alongside the regular pod.
For monitoring the network traffic for the whole cluster, the CNI and/or whatever ebpf-based runtime security stuff you’re using (falco, tetragon, tracee) is usually enough, but I can definitely see the usefulness of subtract for more specific debugging purposes. If run as a DaemonSet make sure to add some pod filtering such as namespace and label selectors (but I’m sure you’ve already thought about that).
> use subtrace in an ephemeral pod for debugging purposes
That's a great suggestion. It'd be like kubectl exec-ing into a shell inside the pod, but for network activity. I think I'm going to prototype this tonight :)
> pod filtering such as namespace and label selectors
Yep, Subtrace already tags each request with a bunch of metadata about the place where it originated so that you can filter on those in the dashboard :) Things like the hostname, pod, cluster, AWS/GCP location are automatically populated, but you can also set custom tags in the config [1].
[1] https://docs.subtrace.dev/tags
Might be an overkill to use it in this capacity but SUSE NeuVector (https://github.com/neuvector/neuvector) can build a connectivity graph between pods and allows to perform packet capture to PCAP. It works at the virtual switch layer and can capture all traffic.
or, there's you know... wireshark...
Side question. Why not do a Launch HN instead of Show HN since you are backed by YC ? I thought YC companies can do a launch HN?
They can, but for open-source startups I recommend they do a Show HN first, for the following reasons:
(1) If the Show HN succeeds, then they can save their Launch HN card to play later; if it doesn't succeed, they can go ahead and do a Launch HN soon afterwards.
(2) Show HN is more about the project/tech, especially when it's open-source. Launch HN is more about the startup/business.
(3) Nowadays many startups offer an open-source (or open core) version for free, then make money with a hosted version. For these, it's a good fit to first do a Show HN about the open-source offering, and then later do a Launch HN about the cloud product.
There needs to be a new term for this grassroots astroturf... Grasstroturf? Get the nerds' attention with free tech, down-playing the fact that it's a business, so later they can pump them for money easier. The pig doesn't want to see the knife, they just want the free trough.
FWIW, I've spent time at YC open-source events where later-stage founders advise earlier-stage ones, and this isn't the kind of advice they give. In fact they would describe that as a terrible strategy and tell startups to avoid it.
If I were to give an example of actual advice I've heard, using the terms of your comment, it might be: keep the nerds happy with free open-source software that actually works, then charge money for things that companies (especially enterprise companies) actually prefer to pay for.
The "paid hosted offering" pattern is the most common of these, since nerds might want to run their own instances but many IT departments do not.
please add YC to the title