Very interesting. I worry that if I use your cloud, and a lot of other people do, all of your IP addresses will get banned by all the big players. It will definitely be a fun cat and mouse game!
Related story: Way back in the day, PayPal was just getting started, and decided eBay transactions would be their perfect customer. The only problem is that eBay didn't allow scraping. So they built an entire proxy infrastructure to go around eBay's rules and scrape them.
It worked. It worked so well, eBay bought PayPal.
The side effect of this is that I got control of the PayPal proxy infrastructure since I was on the security team for both eBay and PayPal after the acquisition.
We used that proxy farm to scrape the rest of the web looking for fake eBay sites (because they would block traffic from eBay's IPs) and we had the guys who built it help us build proxy defense for eBay and PayPal.
So this could work in your favor, if you manage to constantly scrape a large target who might want to buy you. :)
That honestly sounds so fun! Who better to build your defences than your attackers ahaha
Tying into your concern, keeping IPs fresh and high quality will definitely be a balancing act as we get bigger. It'd be one today too if we were to try to offer super granular location controls because there's only so many proxies in X state, let alone X city. Currently, we get to aggregate & QA from multiple proxy providers, so our total pool is 300M+ IPs in the US and so far we've had a 99.95% rate of getting a fresh IP address in a session. So so far so good :)
As for that last point, I guess we'll see what the future has in store for us :P
Hello Hacker News! We’re Nas and Huss, co-founders of steel.dev (http://steel.dev). Steel is an open-source browser API for AI agents and apps. We make it easy for AI devs to build browser automation into their products without getting flagged as a bot or worrying about browser infra.
over the last year or so, we’ve built quite a few AI apps that interact with the web and noticed - a. it was magical when you could get an llm to use the web and it worked and b. our browser infra was the source of 80% of our development time. Maintaining our browser infrastructure became its own engineering challenge - keeping browser pools healthy, managing session states and cookies, rotating proxies, handling CAPTCHA solving, and ensuring clean process termination. We got really good at running browser infrastructure at scale, but maintaining it was still stealing time away from building our actual products. So we wanted to build the product we wish we had.
Steel allows you to run any automation logic on our hosted instances of chromium. When you start a dedicated browser session you get stealth, proxies, and captcha solving out of the box. We do this by exposing websocket and http endpoints so you can connect to these instances with puppeteer, playwright, selenium(in beta), or raw CDP commands if you’re built like that.
Behind the scenes, we host several browser instances and route incoming connection requests to one of these instances. Our core design principle was to allow for every session to have its own dedicated browser instance + resources (currently 2gb vram and 2gb vcpu) while still allowing for quick session creation/connection times. Our first thought was to have separate nodes running in a Kubernetes cluster, but the cost of hosting warm browser instances would be expensive (which would be reflected in the pricing), and the boot times would be too slow to handle the scale that some customers required. We got around this by deploying our browser instance image on a firecracker VM, taking advantage of the lightning-fast boot times and ability to share a root FS.
Today, we’re open-sourcing the code for the steel browser instance, with plans to open-source the orchestration layer soon. With the open-source repo, you get backwards compatibility with our node/python SDKs, a lighter version of our session viewer, and most of the features that come with Steel Cloud. You can run this locally to test Steel out at an individual session level or one-click deploy to render/railway to run remotely.
We're really happy we get to show this to you all, thank you for reading about it! Please let us know your thoughts and questions in the comments.
Very interesting. I’m not sure I immediately see your application either however I have been having similar thoughts.
After playing a popular indi game (Kenshi) I was wondering about the very simple automation interface the game relies upon. Why not a virtual world (with interfaces attaching any external source) in which business logic agents interact through the available interfaces of the environment, and other agents. Though tbh, I imagine the entire environment as implemented in layers of YAML style schemas and profiles. So all data, whether in a datastore, active instance, streamed or serialized can be related to in the same way. An envelope with attributes and content specified by the type attribute. The only code would then be the rendering environment, and whatever these agents call for stream processing.
Sort of a gamification of automation, though what can’t be beat is dead simple account of what any one thing is doing at a given time.
The concept of agents running wild with only schemas of how to interact with their environment + one another is something I’ve thought about a lot. With current models, I guess this would just be function calling or forced JSON outputs, but I think it would make for some very interesting results.
With Steel, we’re really just providing a way to do something similar to the “rendering engine” in this scenario but with the web. So agents can interface with their environments (websites) very easily and at scale (which comes with its own difficulties wrt deloying/managing)
We don't compete with Puppeteer/Playwright/Selenium but we're focused on solving the problem of hosting Chromium browsers in the cloud.
The workflow jump we've seen ourselves and from customers is that it's pretty straightforward to build these scripts locally, but the moment you want to run them in prod, you now need to deal with a ton of overhead around packaging up to docker, resource management, scaling, etc. We want to make that trivial :)
Yeah, that’s totally fair! Not everybody will. In our experience, the problems we solve are the creeping kind that develop over time and show themselves as maintenance costs.
Let’s say we want to add web automation to our app, and we've set up a single instance of Chrome hosted on a server somewhere that we connect to and drive with Puppeteer. Great, now we need to ensure incoming requests/sessions are managed properly (>1 active session). If we have a ton of active requests, this becomes annoying because now the resources on our server are being eaten up — so we would need to scale horizontally (or vertically) but now that comes with potential downtime and cold start issues. If we decide to keep this instance of Chrome running 24/7 then we need to bake in resource management to handle memory leaks and connection issues. If we don’t keep them alive, then this comes with significant cold start times (10s+).
Now, let’s say we want to support a real-time scraping use case that requires multiple browsers in parallel. At this point, we would need to use something like Kubernetes with warm pods or maybe lambdas/cloud run. But even those have their own set of challenges/costs. Managing Kubernetes can get complicated and expensive quickly, especially if you’re not already using it for your app. On the other hand, serverless options like Lambda or Cloud Run introduce latency issues (cold starts) and often don’t provide the flexibility you need for long-running sessions or custom configurations.
Then, beyond the infra, we'll probably need to build capabilities to not get stopped out by anti-bot measures like proxy mgmt, fingerprint rotation, captcha solving, etc.
Pretty quickly, as you grow, a small-scale project can turn into a pretty big maintenance project. Building parsers/agents/scripts is hard enough, so we're hoping to make the infra side as easy as possible.
haha, I love this analogy! If we can do for deploying web automation to prod what Vercel did for deploying front-ends, I'd be a happy camper. With even friendlier pricing of course :P
will definitely check this out. i saw the pricing model on the site, what's the motivation around being open-source here if you're providing the infra free of charge?
Yeah, good question! The reason we went open-source is really about transparency and flexibility. We want people to trust what they’re using and have the option to self-host if that’s what makes the most sense for them. Open-sourcing the browser API also lets us build a better product with input and contributions from the community—it’s a win-win.
As for the infra, it’s not totally free—our managed service is there for anyone who doesn’t want the hassle of hosting and scaling everything themselves (although it does have a generous free plan). Going open-source just gives people the choice: run it yourself if you want full control, or use our managed option for convenience and scale. It’s all about making browser automation more accessible without forcing people into one path.
Very interesting. I worry that if I use your cloud, and a lot of other people do, all of your IP addresses will get banned by all the big players. It will definitely be a fun cat and mouse game!
Related story: Way back in the day, PayPal was just getting started, and decided eBay transactions would be their perfect customer. The only problem is that eBay didn't allow scraping. So they built an entire proxy infrastructure to go around eBay's rules and scrape them.
It worked. It worked so well, eBay bought PayPal.
The side effect of this is that I got control of the PayPal proxy infrastructure since I was on the security team for both eBay and PayPal after the acquisition.
We used that proxy farm to scrape the rest of the web looking for fake eBay sites (because they would block traffic from eBay's IPs) and we had the guys who built it help us build proxy defense for eBay and PayPal.
So this could work in your favor, if you manage to constantly scrape a large target who might want to buy you. :)
That honestly sounds so fun! Who better to build your defences than your attackers ahaha
Tying into your concern, keeping IPs fresh and high quality will definitely be a balancing act as we get bigger. It'd be one today too if we were to try to offer super granular location controls because there's only so many proxies in X state, let alone X city. Currently, we get to aggregate & QA from multiple proxy providers, so our total pool is 300M+ IPs in the US and so far we've had a 99.95% rate of getting a fresh IP address in a session. So so far so good :)
As for that last point, I guess we'll see what the future has in store for us :P
Looking interesting, will definitely give it a go.
Btw, there is inconsistency between pricing page and pricing on docs.
Pricing page for developers is $59 Pricing in docs for developers is $99
awesome - let us know how it goes!
great catch on pricing mismatch, just fixed, thank you :)
Hello Hacker News! We’re Nas and Huss, co-founders of steel.dev (http://steel.dev). Steel is an open-source browser API for AI agents and apps. We make it easy for AI devs to build browser automation into their products without getting flagged as a bot or worrying about browser infra.
over the last year or so, we’ve built quite a few AI apps that interact with the web and noticed - a. it was magical when you could get an llm to use the web and it worked and b. our browser infra was the source of 80% of our development time. Maintaining our browser infrastructure became its own engineering challenge - keeping browser pools healthy, managing session states and cookies, rotating proxies, handling CAPTCHA solving, and ensuring clean process termination. We got really good at running browser infrastructure at scale, but maintaining it was still stealing time away from building our actual products. So we wanted to build the product we wish we had.
Steel allows you to run any automation logic on our hosted instances of chromium. When you start a dedicated browser session you get stealth, proxies, and captcha solving out of the box. We do this by exposing websocket and http endpoints so you can connect to these instances with puppeteer, playwright, selenium(in beta), or raw CDP commands if you’re built like that.
Behind the scenes, we host several browser instances and route incoming connection requests to one of these instances. Our core design principle was to allow for every session to have its own dedicated browser instance + resources (currently 2gb vram and 2gb vcpu) while still allowing for quick session creation/connection times. Our first thought was to have separate nodes running in a Kubernetes cluster, but the cost of hosting warm browser instances would be expensive (which would be reflected in the pricing), and the boot times would be too slow to handle the scale that some customers required. We got around this by deploying our browser instance image on a firecracker VM, taking advantage of the lightning-fast boot times and ability to share a root FS.
Today, we’re open-sourcing the code for the steel browser instance, with plans to open-source the orchestration layer soon. With the open-source repo, you get backwards compatibility with our node/python SDKs, a lighter version of our session viewer, and most of the features that come with Steel Cloud. You can run this locally to test Steel out at an individual session level or one-click deploy to render/railway to run remotely.
We're really happy we get to show this to you all, thank you for reading about it! Please let us know your thoughts and questions in the comments.
Very interesting. I’m not sure I immediately see your application either however I have been having similar thoughts.
After playing a popular indi game (Kenshi) I was wondering about the very simple automation interface the game relies upon. Why not a virtual world (with interfaces attaching any external source) in which business logic agents interact through the available interfaces of the environment, and other agents. Though tbh, I imagine the entire environment as implemented in layers of YAML style schemas and profiles. So all data, whether in a datastore, active instance, streamed or serialized can be related to in the same way. An envelope with attributes and content specified by the type attribute. The only code would then be the rendering environment, and whatever these agents call for stream processing.
Sort of a gamification of automation, though what can’t be beat is dead simple account of what any one thing is doing at a given time.
Hey, Nas here.
The concept of agents running wild with only schemas of how to interact with their environment + one another is something I’ve thought about a lot. With current models, I guess this would just be function calling or forced JSON outputs, but I think it would make for some very interesting results.
With Steel, we’re really just providing a way to do something similar to the “rendering engine” in this scenario but with the web. So agents can interface with their environments (websites) very easily and at scale (which comes with its own difficulties wrt deloying/managing)
If these instances are shared, how do you segregate login details, sessions cookies, …etc? Are you always running them in incognito?
Instances are not shared. :) Everyone gets a dedicated session with dedicated resources. One session for every machine.
I don't understand what problem you're solving that puppeteer doesn't already solve.
We love Puppeteer and use it to power our API!
We don't compete with Puppeteer/Playwright/Selenium but we're focused on solving the problem of hosting Chromium browsers in the cloud.
The workflow jump we've seen ourselves and from customers is that it's pretty straightforward to build these scripts locally, but the moment you want to run them in prod, you now need to deal with a ton of overhead around packaging up to docker, resource management, scaling, etc. We want to make that trivial :)
I have never had any issues running puppeteer on a server, what problems are you solving that would arise?
Yeah, that’s totally fair! Not everybody will. In our experience, the problems we solve are the creeping kind that develop over time and show themselves as maintenance costs.
Let’s say we want to add web automation to our app, and we've set up a single instance of Chrome hosted on a server somewhere that we connect to and drive with Puppeteer. Great, now we need to ensure incoming requests/sessions are managed properly (>1 active session). If we have a ton of active requests, this becomes annoying because now the resources on our server are being eaten up — so we would need to scale horizontally (or vertically) but now that comes with potential downtime and cold start issues. If we decide to keep this instance of Chrome running 24/7 then we need to bake in resource management to handle memory leaks and connection issues. If we don’t keep them alive, then this comes with significant cold start times (10s+).
Now, let’s say we want to support a real-time scraping use case that requires multiple browsers in parallel. At this point, we would need to use something like Kubernetes with warm pods or maybe lambdas/cloud run. But even those have their own set of challenges/costs. Managing Kubernetes can get complicated and expensive quickly, especially if you’re not already using it for your app. On the other hand, serverless options like Lambda or Cloud Run introduce latency issues (cold starts) and often don’t provide the flexibility you need for long-running sessions or custom configurations.
Then, beyond the infra, we'll probably need to build capabilities to not get stopped out by anti-bot measures like proxy mgmt, fingerprint rotation, captcha solving, etc.
Pretty quickly, as you grow, a small-scale project can turn into a pretty big maintenance project. Building parsers/agents/scripts is hard enough, so we're hoping to make the infra side as easy as possible.
So this is the Vercel approach to web bots? Hopefully less predatory aha..
haha, I love this analogy! If we can do for deploying web automation to prod what Vercel did for deploying front-ends, I'd be a happy camper. With even friendlier pricing of course :P
will definitely check this out. i saw the pricing model on the site, what's the motivation around being open-source here if you're providing the infra free of charge?
Yeah, good question! The reason we went open-source is really about transparency and flexibility. We want people to trust what they’re using and have the option to self-host if that’s what makes the most sense for them. Open-sourcing the browser API also lets us build a better product with input and contributions from the community—it’s a win-win.
As for the infra, it’s not totally free—our managed service is there for anyone who doesn’t want the hassle of hosting and scaling everything themselves (although it does have a generous free plan). Going open-source just gives people the choice: run it yourself if you want full control, or use our managed option for convenience and scale. It’s all about making browser automation more accessible without forcing people into one path.