The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see.
A second side effect of a knee-jerk reaction to bots crawling websites is that if you try to fight all bots, you also end up hurting real users that use "bots". If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. That might or might not be what you expected, but it's worth taking into account.
And finally, something worth noting is that there are so many websites whose owners complain about bots, but the real problem is that the website is poorly built and should be improved anyway. Bot traffic is not necessarily bad.
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user.
No; in this case you are not a user, you are a bot user.
I don't disagree, but there is a sliding scale here.
For instance, I wanted to buy a piece of equipment the other day from a local company for a specific usecase. I wanted to find a specific price/weight/specs ratio, and asked an llm to loop through the 20 or so items, fetch their page and calculate and present some values for each. This then led to me going and buying the one I found. So the llm was mainly just an extension of me clicking into every page and making a spreadsheet myself.
However, if it were to continuously poll, or just scrape or something with no intention of buying, I would be no better than a bot.
I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. They clearly don’t mind so what’s the issue? You still used a bot.
> I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you.
The point is that not all bots are bad. Assuming as much / implementing policies to that effect won't block all bots but it _will_ block the portion of bots that represent users considering giving you money.
I have a series of bots that monitor various eCom sites to monitor prices over time for big-ticket items I am considering as well as staples/groceries and everything in between. I have this little scrape/ingest pipeline because there's no other way to obtain this data... not even an API that I can pay for access to.
One of the large appliance sellers that I have in the scrape queue has gotten _hyper_ aggressive with bot detection to the point where even my personal head-full chrome instance doesn't always get to load the page. Guess who I will never buy that ~$2000 appliance from.
If this is the case, they probably outright do not want you as a customer.
Significant sales (beyond for excess stock clearance/discontinued items) exist largely to generate some excitement from potential customers and to lure them in to taking a look at the site/store and to hopefully get them to buy some extra stuff that's less discounted while they're there.
The actual on-sale item is potentially losing money for the business or at break-even.
------
A person who's running a series of bots like this to track the exact lowest price on the internet for items over time and to always only buy at the lowest price it's ever been sold for....is probably the least likely kind of customer to buy anything extra that's actually profitable for the business.
> If this is the case, they probably outright do not want you as a customer.
Almost certainly! No eComm site wants the cautious shopper that's done their comparisons and knows price trends; they want the impulsive people with a line of credit.
I doubt that my local grocery store is happy that I index all of my receipts so I know what I buy, how often and when the prices change so I know when a discount on something I buy every once in a while is actually a discount.
When I built this a few years ago, I had to write a lot of code to make all this work.
I would expect that more and more people will discover that their LLM of choice can do this, that their LLM of choice can help them comparison shop and do analysis of when - historically - is the best time to buy that new TV or whatever (yes, pricing on this stuff is seasonal!).
Maybe _most_ people will never do this, maybe I'm wrong. But if i'm not, blocking all bots "because bots are bad, mmmkay" is how you loose a sale.
Ignoring the specific price data example, more and more and more people _are_ browsing the web through an LLM. Assuming that continues, blocking bots is going to have the same effect on your traffic as doing everything in the "what not to do" section of the google SEO guide book.
The tradeoff here is a classic false-positive vs false-negative issue. If the cost of the bad-bot false positive (which blocks your bot out) is lower than the cost of the bad-bot false negatives (which allow bad bots in), then it is still a good tradeoff, if a suboptimal situation.
you know what a browser is called by the web site? check the header that tells the version.
USER AGENT
not user, an agent on behalf of the user.
the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT.
for a glimpse at one possible futureverse, check out what the home assistant community is up to, how they assemble then update the ambient information displays on their walls. their USER AGENTS are doing what most HN-style "hackers" dreamed of reading scifi as kids.
the alternative is all your in home information owned by corpos when the info should be from user agents not corpo feeds. if you want to vote this idea down, you might be a corpo. :-)
Yeah it's sort of weird that people are so used to using their computer to interact with other computers across the internet that the whole thing has become invisible to them: They completely forget that they're using a computer at all. Along come "bots" and suddenly everyone remembers that the whole thing involves computers. Driving the computer by hand for hours was never efficient, but is marginally entertaining I guess.
This reminds me of all the confused comments that pop up whenever someone calls plain HTML with forms and links a "REST API." "But APIs are for programs! People visiting websites aren't programs."
I fancy the approach myself... but, much like the grand ideas of OSI in networking, Tim Burners Lee's idea of the Web was never what was going to be used and a lot of the original terms/concepts just plain don't make sense for the Web that exists. It was, originally, a true document system, nothing more. It wasn't until many changes from the original idea that it started to be what people wanted & actually use today. Nor was it even part of Tim Burners-Lee's original concept that 99% of Web traffic would be something other than users viewing & editing HyperText documents.
The User Agent would get a HyperText document from a Web server, render it, and show it to the user. You could parse HTML, display images, and follow links - with the assumption the user would want to edit them like any other document. There was no encryption. There was no scripting for dynamic content. Styling was done via semantic markup. There wasn't even any POST when the User Agent term was initially applied. Everything was supposed to be like one big distributed document store instead of centralized hosting on few servers for many client devices.
All that is to say: Regardless what I think sounds nice myself, if the web was just the original vision & definitions instead of everything it changed and grew to become we wouldn't even be using it to have this conversation.
Interestingly, people seem glad to have telephony companies filter bots even though the original idea of the telephone did not call out robocallers as invalid clients. Probably because, unlike the modern Web, the average person gets a chance to see what it means to allow anything inbound even though the original concept never considered robocallers.
I was on the internet before HTML and long before Mosaic...
I agree with one aspect of the narrative you're saying, his idea was held as being for «documents». To be sure, document was more abstract than document on a Mac.
Either way, I'm very sad it mostly turned into animated brochures instead.
But the reason styles were separate, the reason everyone talked about semantics, was always about the universal knowledge DAG and enabling machines to help us navigate and retrieve it. It's why so many of the early agents referenced navigation (Spyglass, Navigator) or the interconnectedness of things (Mosaic).
So, ok, big document “store”, but that's a lifeless noun. The information you are looking for, the dots connected for you surfacing what you didn't know, that's a living growing thing, an agentic retrieval savvy knowledge engine, distributed for curation and resilience.
The librarian was always to be a machine or many, on our behalf.
> for a glimpse at one possible futureverse, check out what the home assistant community is up to, how they assemble then update the ambient information displays on their walls. their USER AGENTS are doing what most HN-style "hackers" dreamed of reading scifi as kids.
I would love some pointers to read up on this, what are some examples of this? Genuinely, not being a troll or ironic
> I would love some pointers to read up on this, what are some examples of this? Genuinely, not being a troll or ironic
There's a "look what I built!" thread every few days on the /r/homeassistant sub and the home assistant forums, too.
The data sources vary but it's usually a mix of devices that the user has locally and some remote data sources like weather/public transit schedules APIs or RSS and in some cases also utility rate info all integrated into a SPOG that is then displayed on some ambient screen (usually a hacked tablet or other cheap ARM device)
It's an interesting example. People did very similar things during the "desktop shell modification" scene in the early 2000s, by far the most common thing people modified their desktop shell to do was display a bunch of charts and graphs of semi-random information, usually about their local computer.
It was useless in 99.9% of the cases. You don't need or want to have constant access to how much free ram your computer currently has, as if that even meant anything.
So, after a few steps, that leads to the question "what is useful information? How do you define that?"
A basic attempt at it might be: useful information is something that causes you to make or change a previous decision. Take an action you wouldn't have otherwise done.
If you're living in the american southwest, and it's between the months of like, march and, dunno, septemeber, you probably don't need to know the current outside temperature to decide if you're wearing shorts or a jacket when you get up in the morning (you're wearing shorts because it's always hot)
Theoretically there could be some kind of "extreme" weather, aka rain, that causes you to do something different, bring an umbrella, but it's pretty difficult to not notice it currently raining, even if you're inside a house.
So what other information do you want? There's all sorts of stuff that's considered "news", but I'd argue that at least 98% of that is basically just entertainment. We'd need some kind of article or perhaps even a book to hammer out all the nuances involved, but seeing a headline in the morning that trump has caved in to iran again or killed some more random people on a boat, while sort of important to know, doesn't really change what you're going to be doing that day.
Knowing ICE is trying to do some domestic terrorism near where you might be could be useful information, but there's not a real good way to track that.
So yeah, what kind of information is actually useful to have in the morning? Stuff like calendar appointments would count, knowing you have things scheduled at specific moments or tasks to do, go grocery shopping, etc, todo lists in otherwords, but what else? That's basically all self-generated stuff by you about you, you don't really need any outside data sources.
> Stuff like calendar appointments would count, knowing you have things scheduled at specific moments or tasks to do, go grocery shopping, etc, todo lists in otherword
So ... most people.
> That's basically all self-generated stuff by you about you, you don't really need any outside data sources.
Other than the ones you just enumerated that most people are likely to use, I need to know - using external sources:
- When to expect the garbage truck (do I put the recycle out tonight, too?)
- Is it a school holiday?
- When it's cheapest for me to charge the car up; if there are any "grid stress, please turn non-essentials off or we're getting rolling blackouts" alerts.
- Same goes for nearby fires or other issues that are going to change the "when do you open/close the windows and switch to AC" math.
- If it's likely to rain in the next few days so I change what plants get watered / when
- Packages that I am likely to get in the post today
- Weather (for display, not just for use in the automations mentioned above)
- Where my power provider is currently getting its power from (so I can calculate estimated CO2 emissions, if any)
And that's about 1/2, probably. Can't be bothered to go spend 20 min pouring through my HA config to build the full list.
I also stream the npr newscast as an MP3 as part of the "good morning, while you wait for coffee..." automations, if that counts for anything.
> you know what a browser is called by the web site? check the header that tells the version.
You mean that key whose only legitimate values also call every browser Mozilla? Arguing from the dictionary is so tedious. I'm going to scrape your site, but the idea that you have some obligation to like it because of the definition of the word "agent" is profoundly stupid.
A lot of people live off ads. I block ads. People who live off ads aren't obligated to like it - and are free to take any means to block it without moral hazard.
This website refers to browsers as browsers. Just because a header in the underlying protocol is named a certain way, that doesn't necessarily reflect what the humans intend.
I downvoted it because you don’t use capital letters and still feel inclined to make people read this rant. Corpo is a really stupid slur and reading this rant was a waste of my time.
So instead of adding of addressing his point about User Agents, you decide its better to make a low level bullshit comment about the very last sentence to stand up for... checks notes... Corporations and dismiss everything else. Top notch quality content that for sure added to the conversation.
"Bot" access on behalf of users should be fine, even preferred. The missing piece is some micro transaction layer and some sort of attestation somewhere in the layer that the person driving the bot is not a bad actor.
Equating bot with bad actor in 2026 is Luddite behavior. Driving people to your site so you can serve them adspam or just make whatever operation they want to do 10x more difficult is the same.
MCP style APIs should eat the web. This doesn't mean the "open" web goes away. > 99% of the time I don't care for a domains particular FE at all. It's a complete waste not only of time, but resources and bloat. Pushing the contract into the agent should become good UX. Making things harder for good faith users should never be the goal.
The best way to read the information on the internet today is via a LLM.
Just like you don't process raw data in a data warehouse or datalake by looking at tables, you use a SQL or BI tool, to process all the information on the internet you need to tool to digest it for you. For many, today that tool is a LLM chat interface or agent.
I disagree. I personally consider these my biggest problems with the Web:
- Bias, specifically commercial bias
- Webpage formatting: every website looks different, hides the information I want in different places. Sure, it "looks pretty", but I don't want pretty; I want info
- Scams/SEO/etc...
The LLMs are very good at reading from multiple sources, parsing, and presenting only the data in a consistent format. There are many examples of this, but if you want a good one to try for yourself:
Google "How to make ham fried rice"; you'll get 10,000 articles, most pretty good recipes. But they're all different; most of them are just bait for ads. And most of them, the 10-line recipe is hidden between 50 useless paragraphs about how serving food is life's most important goal.
Now, ask an LLM to search for it, find the best combination, and list only the recipes. You get a perfect, 10-line recipe that doesn't waste your time.
I personally prefer to read a couple of recipes to understand the context, what people have tried, what they thought of it. Only when I have a good general idea of the recipe I choose the best one I found based on difficulty and the ingredients I have. I then do the plate, often not following entirely the recipe I chose, and after that and eating it I write down the name of the plate, the recipe link, the date, and a comment about what I did and how it went. Cooking is a lot more than executing a list of steps like a robot.
Why even use a LLM, just buy a cookbook and you can even waste less time and know the information is valid. And again if your logic is consistent, google routes your search to an LLM automatically and renders the reply - that will save as much time and do the same thing since LLMs are so good at parsing and presenting information.
So why should I pick an LLM over google's inbuilt LLM or an actual cookbook on the desk? What wasted time is the LLM saving?
Maybe, but it's reality. Not too long ago the rule would be "if it doesn't show up in google, it doesn't exist". Today agents+LLMs are better browsers than Chrome/Safari/Firefox/... If a webpage does not show up in an LLM it may as well not exist.
Because the economic, social, and climate impacts are at best uncertain and at worst devastating to the majority of human beings on the planet. It's always surprising to me that people don't intuitively separate the usefulness of AI from its risks. It feels like everyone has to be a doomer or a booster.
Because AI cannot be trusted as a reliable source of information. Information is more trustworthy when it comes from from primary sources on the open web.
Why not ?
Can google be trusted ? Or facebook ?
How do people have been accessing the internet for the last decade you reckon and how's this any different ?
Google (as it originally) existed wasn't a source of information, it was an *index* of it. You were trusting google as a source for {search_query}, you were trusting the links it gave as a source (based on your own evaluation). LLMs are fundamentally different, because you are trusting the software to actually generate the information in a truthful way.
If you use google (sans AI), you're putting some trust in their page ranking algorithm. If you use it with AI, you're trusting the same algorithm (since that's how the model gets it's sources), but then you're trusting the model to evaluate the sources for credibility and extract the information you actually want.
I routinely run into Claude Opus and Fable making basic mistakes like misunderstanding a simple negation, which would be on top of whatever reliability issues there are with a source. I think that means it is functionally very different.
No. Neither can be trusted as far as you can throw them. They're both incredibly invasive data brokers. Their customer facing products are just vehicles to show ads and Hoover up more PII and behavioral data on everyone.
I think it's time for people to build a local database of every site they've ever visited, and then they can give their LLMs access to that. I might be willing to pass this data off to a company to store for me. Companies already store all my emails and money--why not trust them with this too?
Like, all the comments of this post would go into my personal database simply because I loaded the page, and it would help me find old information I've read, and could also inform LLMs I use.
There is bot traffic initiated by a human and bot traffic not initiated by a human. I would want to serve the first but not the second if it impacted my cost/performance at all.
Isn't all bot traffic ultimately initiated by a human? Someone plugged the computer in and gave it instructions. It may result in one http request or billions of them, but the human is still the initiator.
Then you are a uninvited bot that crawls my content in order to send alarms to people to tell them to visit my site?
This is a "what if it's for your own good?" argument. What if I break into your house to clean your toilets? Although amusingly, if you're a "web alerts company" that I didn't contract and sends out alerts in batches on your own schedule, you will probably send a ton of your customers to my site at the same time and slashdot me off of the web entirely.
No, I won't send a ton, I'll send a few dozens to a few hundreds at most because not everyone is interested in the same things. And they'll visit at their own timeline. If your site can't service a few dozen requests simultaneously then you probably aren't a news site in the first place so the whole argument is moot.
because they maintain the websites for social status, if the user never sees the specific website or knows they used it, you can't gain any social status
Are you implying that's a bad thing? Doing things for social status has been an integral part of society for millennia. It's a legitimate motivation that can benefit both the recipient of the status and the rest of society. In this specific case, if you assume that the author makes content that is useful to you only because of the social status reward, taking away that reward means the author will stop making content that is useful to you.
One more reason for the bot difference is the llm bot users are doing almost all the traffic, and it feels wasteful, painful, and there are reports of 99.x% being llm bots just hitting webpages over and over pointlessly.
This is a change. People put up new caching layers, and it urks people running a website in a single small machine. Yes, this could always happen with slashdotting but it's different now.
The social contract of accepting scraping for visibility was always tenuous and is now fully dead. But it’s shocking the number of people who are essentially victim blaming here. You should spend YOUR time to optimize your free website so MY use case is not impacted. How is that not incredibly selfish on its face?
The social contract for putting a website on the public internet involves following standards and making your website accessible. You SHOULD make sure that your website is readable to people no matter what user agent they are using as long as it's standards compliant.
For example, making a website that only works in Edge and refuses to load for anything else would be a shitty practice. You are free to do it. There's no one stopping you, but you should expect people to bitch about it and they might think you're kind of a jerk or maybe just bad at making websites. It's the same thing when a site actively rejects traffic from curl, wget, or any other common utility that many people find useful.
The social contract for putting a website on the public internet was built on the assumption that browsers would be used to access those websites, and that most browsers would generally behave in a similar way (click on page, display page, pretty simple).
LLMs are not browsers, nor are they people, so I don't see why they should be part of this equation at all. They do not display the page for a real person to read, they merely ingest its contents and then spit out something completely different.
> The social contract for putting a website on the public internet was built on the assumption that browsers would be used to access those websites, and that most browsers would generally behave in a similar way (click on page, display page, pretty simple).
No, not browsers, user agents (some of which would be browsers) and there was never an assumption of how a website would behave on the user's end, that's the job the user agent. All that HTML and CSS are only suggestions, but the power was always intended to be left to the user to decide if/how they wanted that data presented to them and it was always intended that the user be able to choose whatever tools they wanted to collect, process, and display content pulled down from the internet. That same principle is how we have ad-blockers. You are free to infest your website with ads, but as the person requesting the website I'm under zero obligation to display any part of that site I don't want.
> LLMs are not browsers, nor are they people, so I don't see why they should be part of this equation at all.
LLMs are just another tool used by people to collect and process the information available on websites. Maybe there is a distinction to be made between people using LLMs to get web content and corporations scraping websites to take training data, but even scraping has always been a common and expected practice. It's the current scale that is making things different.
It's probably a pretty common misconception that websites should have total control over how their content is viewed, but it's strange to see that on a website like this. Part of the reason apps are so popular with companies is that it gives them control over how content is presented which is something webpages were never intended to give them.
Exactly and Cloudflare (there should be others out there too - open source options) has proposed that AI bots need to pay for access to our websites. If they and or others could pull off blocking AI access until it pays creators then AI is forced to pay as it should and always should've!
Overall systems and different business models need to be created that forces AI to pay it's fair share! Elon says there will be an abundance thanks to AI and we wont have to work -- ok how's that gonna work without different systems in place paying us?
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user.
People trying to block bots end up keeping out all kinds of users. I get blocked frequently for using a regular browser, just with JS disabled. 99% of the time, I just close the browser tab and move on with my life.
You take your site off the public internet and paywall it off. Websites on the open internet should be accessible and ideally the goal should be to share something cool with the world, not just to make yourself rich.
This. The Internet was a better place when people hosted things they wanted to share at home or on their ISP's servers, without the expectation of making money.
Paywalling off the site solves the problem of monetizing bot traffic, any bot crawling your pages paid you to be there, but it can't fix the plagiarism/copyright infringement problem
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user
You are mistaking yourself, well your bot, as his target audience.
You might as well say “If I want to send you my commercial email, and you block it, you hurt me, the email user.”.
While your point of being concerned about cloudflair becoming a global arbiter of who gets in and who does not (which may at times not just mean blocking bots, intentionally or through technical issues), the need to block the deluge of bot traffic is very real for many sites and that is one of the easy options for them to deal with that. There are other methods like directives in robots.txt and nofollow attributes on links, but so many bots simply ignore those that they are not really useful.
> Bot traffic is not necessarily bad.
Nor is it necessarily wanted. In fact, it often isn't. Unfortunately practically all bot runners seem to either assume that their traffic is the special good kind or not care either way.
> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.
I WANT! I WANT!! I WANT!!!
Well, that site runner wants you to access the site as a human, if at all, not via bots. Sorry to be the one to break it to you, but what you want isn't always the most important factor for the rest of us.
> but the real problem is that the website is poorly built and should be improved anyway
Firstly: just no.
Secondly: if you and your bot don't like our badly built sites, feel free to go get your information from those that you consider to be better built. Problem solved.
> If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse.
If the the processing is subject to GDPR (e.g. if controller is in EU) then you do have recourse. You can complain to DPA or sue the company. The company is ultimately responsible for the decision to block you, at least in cases where you personally tried to access the site.
And if you believe that blocking bots is bad, then by all means provide a better alternative that doesn't increase costs for sysadmins. If you believe that the website should just be improved, then by all means feel free to provide instructions on what should be improved and exactly how so that we don't ever have to block bots anymore. I'm sure all the sysadmins having to deal with issues like this one will thank you
This is a whole lot of things you want and feel entitled to. Nobody has to cater to you - we can block whatever we want to block. And if our poorly designed sites bother you, that’s too bad for you. But your wants are not my problem. If you want someone to cater to you, pay them. You aren’t entitled to anything.
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.
The article you're replying to describes in explicit detail how the bots and their operators have directly caused and continue to knowingly cause real harm to the author and others in similar positions, both financial costs and administrative/maintenance burdens that would not have otherwise been required.
You then respond "but if you block the bots then I won't be able to use the bots, and that harms me because I might have to read your web site myself..."
Are you serious?
> That might or might not be what you expected, but it's worth taking into account.
I would wager that for almost everyone who is blocking bots after getting functionally DDoSed by them this is absolutely an expected and desired outcome.
> And finally, something worth noting is that there are so many websites whose owners complain about bots, but the real problem is that the website is poorly built and should be improved anyway.
Both can be true. If you operate a git repository with a public-facing web interface for example there are going to be a lot of possible operations that are inherently expensive but also incredibly rarely used by normal users so it doesn't really matter, but the bots now ignore your robots.txt and are programmed to go after every link they can find, so they trigger every single possible expensive operation more times in a night than your actual users ever have in the history of the site while dividing requests across so many different IP addresses that rate limiting becomes impossible at the individual scale. These days they're even feeding the discovered URLs back in to their models to have them invent new possible URLs and trying those in hopes of finding content never publicly linked. They will send you thousands of requests for URLs that they literally made up.
Sometimes the site is in fact badly coded and operations that should be simple have higher costs due to bad design but you don't have to look very far to find situations where legitimately high-cost resources are exposed to the public because they're expected to be used in a non-abusive way. We should always be standing up against abuse of public resources, unless we want to lose them altogether.
> Bot traffic is not necessarily bad.
You are right, but whether it's good or bad more or less comes down to a cost/benefit analysis. As we've already covered infinite times, these bots being used to train LLMs cause significant real costs to the operators of these sites. What benefits do they offer in return? We know the clickthrough rates are terrible, so what other reasons would site operators have to make those real costs worth it? So someone can get a response back from an algorithm that confidently misinterprets or even entirely misreports what the data actually meant?
Even the most die-hard "information wants to be free" types who absolutely want their datasets trained on would probably prefer that the bots accessed the data directly via an API or downloaded a database dump rather than spidering and scraping a web interface intended for humans.
> the real problem is that the website is poorly built
This is almost always the problem.
This is currently the problem that GitHub is having too. If they had remained with their crusty old rails architecture, very little of this mess [0] would be occurring right now. They would have been able to focus all their engineering talent on scaling the product rather than inventing elaborate client side state synchronization mechanisms.
We tried to deploy private cloud github enterprise node back in 2018. It was pure garbage without CI integration. I am happy microsoft took harder long term decision!
Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software.
How do you figure it hurts real users? The amount of compute/energy used on the proof of work is pretty minimal. You're using more when you watch a YouTube video or browse a JS-heavy web app.
Of course a sophisticated scraper can "figure out" how to bypass. It isn't trying to be foolproof, it's adding an extra cost to deter massive amounts of bot traffic.
I put it in front of my hobby project because I can't afford to serve hundreds of thousands of bot requests from residential proxies all across the world, and I didn't want to route all of my traffic through a third party company like Cloudflare.
1. It makes the web more annoying and time wasting.
2. I'm a real user and I have to enable JavaScript and run arbitrary programs from untrusted websites to access them. One browser that I use doesn't even support JavaScript (w3m). Fortunately it is unaffected because it doesn't have the string "Mozilla" in its UA and Anubis lets every user without that pass to not hurt legitimate bots like search engine crawlers. As said Anubis is trivially bypassed. The only reason it isn't bypassed is because it's simply too irrelevant to the AI companies.
I empathize, I hate the bloated JS-heavy web and avoid it when I can. I just don't see a better solution. The options for me are: sacrifice some legit users by using Anubis, or sacrifice all users by shutting down my service because I can't afford to serve the bot traffic.
There are false positives. Every time I've used a tool like this, I get reports from my users about it blocking their access. It's especially bad for users who use privacy oriented browsers like Brave or use VPNs.
For a personal site where you don't care about 0.1% of your users getting blocked, it's probably fine. For a business, blocking some of your paying customers isn't acceptable.
> For a business, blocking some of your paying customers isn't acceptable.
That depends on your business and the costs you're incurring from the bots. It might be justifiable to lose 0.1% of your customers if it means saving server costs from serving useless bot traffic.
I suppose it would depend, yeah. I find having paying customers being blocked to be completely unacceptable. Not only is a bad customer experience, but they tend to be very vocal about it. Posting online, leaving bad reviews, etc.
Most of the friction is just JS overhead for the computations, a compiled solver is like 1000x faster. If Anubis ever gets popular enough that scrapers care, it would be trivial to defeat. And last I checked you could bypass it by just modifying the user agent
I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.
It's actually a genius idea. If you are someone who the professional presentation of not having an anime girl on the loading page is required, then you can afford to fork over the cash to fund development.
I guess the disconnect here is a bunch of HN'ers believing professional companies and websites want to attach their branding to a sexualized anime character and that they are willing to pay to remove it.
Which one then wonders why they would install it in the first place.
> is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.
Most people either don’t know or don’t care about “those subcultures”. I bet a lot of older people think it’s a cartoon figure of Betty Boop (nurse) and miss the furry bit since it appears and disappears quickly. Most people also don’t have a brand.
So it seems like you’re describing a concern that only affects a tiny fraction of people:
- Not interested in paying for custom branding, so obviously not a corporation or influencer
- Dislikes cartoons
- Aware of, and hostile towards, “furry” subculture
That has to be an exceedingly small fraction of potential users of Anubis, and given how much businesses and branders will pay to custom-brand something, I’d counsel them to stay the course. Sure, a few never-payers will never pay, but they wouldn’t have anyways, so they can cope with Nurse Betty or look elsewhere for a competing free product.
If you think about this in physical market square terms — in other words, a bazaar — it seems horrendously rude to complain about a shop logo sticker on a free product handed out to anyone that walks up and asks for it. If you want it white-labeled so you can write your own name/logo on it, you pay for the privilege of displacing their name with yours. But you don’t stand there and loudly complain that their shop mascot has dog ears while holding a freebie bag of product, without losing the respect of everyone who hears you doing so.
I'd argue its rather functioning exactly as intended with regards to obtaining paid users.
Also this view seems a bit elderly. For most towards the end of the millennial curve and younger, anime is no longer subculture, its just general culture at this point. Although I would agree it isnt necessarily what you'd want for every platform and web presence.
How hard is it to maintain a fork that changes nothing except the logo?
If you can't be bothered to maintain a trivial fork, then why should the author of anubis be bothered to serve your branding needs? It's not like you have a service contract or anything do you?
It's MIT licensed, you are free to do whatever you want to it, including removing the logo. They're basically just saying "we'd prefer you didn't do this, but we're not preventing you from doing it."
Someone willing to take the 2 minutes to ask Claude to change the logo can probably also ask it to fix any bugs they find, or add any new features they might want.
Nah, I don't especially like the logo myself when I come across those sites, but I do like that "brand" people who want to make money off it and not pay dislike it even more.
You wish they'd give your preferable imagery for free and not make you feel bad for using their free software for personal gain. Your subculture is irrelevant and not special.
This is like getting angry at cookie banners instead of all the companies tracking and selling your data.
You're complaining about the symptom (needing to have these checks) not the cause (if they don't, 99% of their traffic will be bots, the site will slow to a crawl and be unusable anyway).
In any case, I saw the dumb anime girl for about 2s then the site loaded. Not a big deal.
The GCC bug tracker uses the meta-refresh challenge, which does not require JavaScript. Due to the fact that the server makes sure the client has waited at least 75% as long as it should, the HTML has to add one second to the meta-refresh wait. Patches welcome. Meta refresh granularity is in single digit seconds.
I'm assuming a bot running a headless browser instance can still get past it?
It's still valuable to raise the cost of scraping of course. I don't think anything can really stop a determined scraper from impersonating a human. I wonder though if a system similar to Anubis but mining some crypto would make bots _welcome_ - since they're paying for their traffic.
People tried this in 2013 or so, there's no point to it. Doing proof of work in javascript in a browser is so crushingly, pointlessly slow that there's no value at all. Some browsers also intentionally detect attempts to do proof of work and attempt to block it entirely.
To help you combat these unwelcome scripts, we’ve partnered with Disconnect and compiled lists of domains that serve cryptomining scripts so you can block them directly through Firefox. When you download Firefox, tracking protection is enabled by default, blocking known cryptominers.
Which is trivially bypassed by an actual implementation of the proof of work in non-javascript, rendering it absolutely useless. The website is approximately 3800x times slower than native code, and hundreds of thousands of times slower than the CUDA kernel claude wrote. The "proof of work" is just non existent at that point, they're solved in milliseconds for what would take the browser version 10 minutes or more, it's security by obscurity being dressed up as something more.
pow_server http://127.0.0.1:8080 backend avx512-x16
──────────────────────────────────────────────────────────
uptime 00:03:12
solver ● BUSY difficulty 9, 0.3s
queue [####################............] 5/8 peak 12
──────────────────────────────────────────────────────────
accepted 1240 solved 1180
503 shed 48 504 timeout 2 4xx/5xx 10
──────────────────────────────────────────────────────────
last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16)
hashes 3.90GH total avg 65.3MH/s
Ctrl-C to stop
Claude even made a nice little API server for it after implementing midstate compression, AVX multi way hashing, and a CUDA kernel. This doesn't stop the literal LLM it's trying to block from solving the challenges, it's really annoying that everybody is using it and claiming that it's something that's usable in the real world as a result of it using proof of work. It's obscure, and obscure is fine so long as nobody is pretending that it is secure.
It is not absolutely useless, empirically, which you'd discover if you had a website getting hammered by bots and experimented with anubis as a countermeasure.
While dedicated scrapers/attackers could work around it, and they could do so much more efficiently than the client-side js, almost none of them do. Unless you like paying additional hosting resource fees to serve bots, it's a worthwhile option, and is less annoying to typical human visitors than cloudflare's interactive captcha/challenge which is what most people use.
The main author is aware that the algorithm is far from ideal for this purpose. See https://news.ycombinator.com/item?id=48869064 . If more bots start to answer the primitive challenge anubis uses now, that'll hasten implementation of a different algorithm.
Don't let the perfect be the enemy of the good enough. For now, the algorithm or challenge scheme almost doesn't matter. Since it's much smaller-scale than cloudflare's challenges, that's probably why very few scrapers and botnets bother to solve anubis's trivial sha2 pow.
Targeted attacks may not be repelled at all. That's not the point.
It does not even require the PoW thing Anubis does. I've setup a simple logic that just:
Checks for existence of a specific static cookie, if it does not exist, output a small page that sets the cookie via JS and reloads. Sadly this kills Noscript, but it would be possible to add a form in <noscript> that when submitted sets the cookie serverside.
Is this trivial to bypass? Yes. It still keeps out 95% of unwanted bots.
Reality is most do not target you specifically they just want to mass-scrape with low effort. Running headless browsers is way more expensive for their op
I've extended this with a FCRDNS checked exclusion for Googlebot.
Another quite effective measure I figured out was checking the existence of Sec-Fetch-Dest header if the User-Agent claims to be a modern browser. If you don't want to close down too much.
Also, I only apply these rules to routes that are not cheap and cached.
That's not far from what anubis does for clients that are determined to have light souls. It doesn't always send a PoW challenge.
For a webapp that sets a long-lived cookie, that cookie could be used to bypass anubis completely, or lower the weight in anubis so that it doesn't send its pow challenge unless there are major red flags. If bots start to abuse that exception, it can be removed.
I once discover you can put escaped XML or json in css content. The purpose was to have static data sets that work cross domain. No headers to configure no letting strangers run all you can eat malicious js on your site.
Define "scale". For any reasonable wait that you're willing to impose on your users, any PoW scheme heavily favors attackers. They have unlimited time and can be scraping even while they're asleep. Your visitors on the other hand don't have that luxury. You might argue that's not the point and it's only to stop dumb scrapers that are effectively ddosing your site, but if it's just dumb scrapers, you could've stopped them less onerous measures like tls or javascript fingerprinting.
It's more or less designed for it, it's SHA256 with a break in the middle for midstate compression to be effective, and the difficulty system is based on a misunderstanding of how bitcoin PoW works ("number of zeros" is never, ever a consideration in bitcoin, it's a match to a floating point target).
sha256(challenge + ascii(nonce)) means that the first compression round of the function can be cached and the second compression round is just the nonce plus the cache. This is the same trick used in Bitcoin mining and would have been avoidable by putting the nonce first, so immediately any non-naive code has to do half the proof of work as the vanilla solver.
write a implementation of the anubis proof of work in native c code, optimized for speed above all else. use every trick available to make the proof of work as efficient and fast as possible, including modern processor tricks on the x86 platform. your code should avoid using external libraries where possible, include tests, and be readable and concise. a reference for what needs to be met is in this repository. https://github.com/TecharoHQ/anubis
Then
let’s develop this more. turn this solver into a local HTTP server that can be given work in the request, and it returns solved work. make an end to end tester that sends test work to the solver and waits for a valid response. add support for solving with a GPU using cuda.
Then it was done more or less, it happily made a local server that supports solving the challenges given to it in bulk with priority based queue and can tolerate potentially tens of thousands of requests a second with no issue. The CPU time spent solving the challenges is less than the SSL setup for the connections. The GPU version does in excess of 20GH/s (but with high latency) though I didn't really test it, I'm not using this for anything but proving a point that the LLM itself can write the bypass tools and run them happily.
I'm almost ready to ship the wasm feature in the next version of Anubis after the one that's about to come out.
The big blocker is that testing against dozens of googles chrome to ensure functionality on abandoned smart TV oses takes a nontrivial amount of time. As an example of the level of debugging and the like required: https://github.com/TecharoHQ/anubis/pull/1684/changes/67621f...
My office gets very warm when chromesweep runs.
This is something that is complicated enough that even though LLM tools can help, it's not a magic bullet. It's just complicated in general.
> My normal bill for running this whole site is around $90 a month. During one bad spike month, it jumped about 500%.
This is D1 - which has very surprising costs. you may just want to drop D1 and move to a static site. There’s no reason your site should cost this much.
There's a lot of people who host their site at extremely expensive places and then do everything they can to minimise unneeded traffic - instead of just moving to a cheaper host. Vercel is another popular extremely expensive host.
I'm always flabbergasted when I see what people pay and how much effort they need to invest to keep their cloud websites from eating them alive.
My allegedly more complicated VPS stack needs an afternoon of attention every two years when a new Debian major release is necessary, and costs have been predictable for 15 years, no matter what happened traffic wise.
The predictability of costs is underrated too. You don't want your hosting solution to automatically scale up to $20,000. You want that if it's overloaded it's simply overloaded.
Thank you! I need to tighten up my KV compression, which is actually carrying a lot of D1's load otherwise. We also had some bad queries some months, as the pages and database grew, that were counting the wrong things (or extremely inefficiently) and those have since been fixed.
Over the last 72 hours, Claude-searchbot [1] alone fetched ~205,000 pages. Sent exactly 1 referral. There is a lot of free financial data on the site, hoping for real users to benefit from it. It is hard to not feel a little cheated out that Claude gets to claim "Found it!" to its users without me getting no credits or compensation whatsoever.
[1] Exact user agent `Claude-SearchBot/1.0; +searchbot@anthropic.com)`
> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.
There's a difference between someone running a scraping tool occasionally and bots constantly and rapidly re-scraping the same site over and over again
Similar to the dose making the poison - the thing that jumped out at me in this blog was the ratio of scraping to visits. Unless OP is scraping thousands of times a day I don't really think they're in the same class as the bots they are blocking.
Something suspiciously absent from this article is addressing whether some of these are in fact human visitors, but humans issuing chatgpt or similar queries instead of going directly.
Is it really a bot if it's in response to a human asking for some aggregate information about charities, triggering a web search and then following the result links to get details for the human? Well, clearly yes it is, but it's a very different proposition from this article's implication that "they have no throttling on their scrapers"[1].
> Challenge 46 datacenter ASNs. Humans don't browse from AWS.
People who have workstations in the cloud do.
> The bots use 99% of the bill and I pay 100% of it.
Running a site this way is always a wallet-DDoS risk.
[1] though yes, by far most will be pure automation with no human in the loop. It's an assumption on my part, but feels like a safe one.
> Something suspiciously absent from this article is addressing whether some of these are in fact human visitors, but humans issuing chatgpt or similar queries instead of going directly.
That's not absent from the article, it's right there in the section titled "The Claude ratio". ChatGPT, Claude, etc. use different user-agents for scraping vs user-initiated requests, and the author notes that user-initiated requests were an absolutely miniscule fraction of the total traffic.
I'm in a similar boat. Probably 99.999% is bots. I have nearly 1m unique "visitors" according to cloudflare and my real users are in the dozens a day. That being said, I love the open internet and am holding on to keeping as much open as I can.
I shutdown all my little informational hobby projects this year that I've tinkered with for decades
They all shifted from mostly paying for themselves (or being so cheap it didn't matter), to essentially producing zero income while resource usage leapt up in magnitudes
I couldn't justify the stress and hassle of making sites, that a few dozen people a day might find useful, into some complex hyperscalable obligations just to feed the bots
It's all OpenAI / Anthropic / Singapore/China based crawlers. I guess they gain data that I provide for free. I get very little crawling by Googlebot / Bing etc which barely even index my site, I only have a single page "indexed" by Google.
Can someone help me understand the underlying motivation behind this?
It makes sense that some crawlers, in the style of Google, would want to index the entire internet. But what is the point of the same crawler re-fetching a page they already fetched an hour ago? Or possibly all this traffic is just independent entities, each trying to cache the internet? The scale of bot traffic makes this seem unlikely.
What's the motivation behind the same entity re-fetching a page it just fetched less than an hour ago?
They are poorly implemented by the "fuck you I got mine" crowd. They will get stuck doing things like trying to run through a calendar that could theoretically go back to the beginning of time and all the way to the end of time. And because that calendar might change, it gets scraped for every inquiry made to the poorly implemented AI system.
I run scripts on my servers on an hourly basis to check which are the top 25 IPs visiting the server (aggregated by /24). If anyone in those top 25 IPs are from China, Vietnam, etc, or from Alibaba/Amazon/etc, the /24 gets blocked by iptables.
It's far from perfect, but it was a quick way to get rid of bots, while not completely blocking people from countries such as Vietnam.
However, on a Gitlab instance I manage (500 users), we have to restrict viewing of git logs and pretty much everything except issues. The bots were too aggressive. Chinese crawlers have access to a huge range of IPs and they often do only 10-20 requests per day, while generating in total over 50k requests per day. Our server load went from 99% down to 0.1% after that (and it's a fairly big server).
My solution was similar. Anything coming from the ASN of a major cloud provider gets a CAPTCHA with a little nuance to allow Google and Bing to index. That seems to do a pretty good job. Also, anything coming out of China or Singapore also gets a CAPTCHA as my site is not popular in those regions and many Chinese bots seem to show up as a Chinese mobile provider. So far, the bots have never attempted to solve the CAPTCHA.
> There's a real conflict here. I want Google and Bing and DuckDuckGo to crawl my site and send me new readers. But I don't want everyone else strip-mining it.
Kinda sounds like we're missing a peer to peer network here.
Instead of downloading the same data over and over again we can just download it once and then share it.
Wouldn't that be better for everyone involved?
It would also function as a distributed WayBack Machine, in case anything ever happens to the Internet Archive. (Which I think is desperately needed, bot apocalypse aside.)
You can validate by repeating the work and sign your content.
The domain name system is now just a rent seeking scheme. It was great as a temp solution but over time it has deleted more content than preserved. I might in theory be billed for having a country name but I don't pay for a city, street name, house number or postal code. Online you should be able to move your widget shop to widget street. Can bill people who want to live on real estate street or used car street and/or set some requirements.
Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.
Are you speaking from experience, or inferring from articles like this?
I serve a static site on the lowest Linode $5/month VPS and it is grotesquely overprovisioned for that use case. It is not the case that every site is getting slammed every second by hundreds of requests per second.
Now, if you have some sort of dynamically-computed website that is generated by a slow scripting language that is poorly optimized and hits the database too many times for a single page, yeah, it doesn't take many RPS to take you out. But that isn't the only option; it's the slowest of the slow options. Realistic, there are plenty of sites that match that description, but I concatenated that many clauses on purpose. Drop any one of them and your personal site will be fine.
It's almost always the DBMS that's the bottleneck when websites drop from traffic. As long as you don't do anything fancier than primary key look-ups you're probably fine though.
I survived handling the search search traffic generated by this thread[1] on PC hardware off a residential broadband connection without any sort of degradation. Only time I've gone offline from traffic was when Elon Musk tweeted a link to one of my blog posts, and that was just a short temporary blip.
For a “personal website” or any relatively simple static content site, the DB should not be getting hit much at all. Cache is your friend. For my sites I have a map of all relevant queries and when we do updates we run the relevant queries one time and cache them to both memory and file system.
Should being the operative word. In practice it's both easy and common to have a database do unnecessary work when generating a page, a problem that often only rears its head when under unusual load.
Yeah, IIRC my django site was 3 queries, all correctly indexed, for a main page hit, and 2 for the actual posts. I don't recall the exact perf numbers, but I'm pretty sure it was easily in the 50/rps range for a small dual-CPU host... which doesn't sound like much in "requests per second" but is enough to cover a front-page-HN'ing just fine. And that "rps" was just the database-backed pages, all the static content was served over nginx, so that 50rps is a "real", 50 humans per second rps, not something getting consumed by only two or three humans.
Static files on literally any hardware from the last 15 years on modern server software simply won't get cooked by bots. The network switch will bottleneck you before the server will. Your ephemeral port range will run out before the server will.
I had a client ask me to look into why their bill spiked 5x (admittedly not by much - from a few dollars to tens). They run a static site, and it turned out someone had inadvertently replaced an image with a massive uncompressed photo. Every unique visitor was caching O(25MB) when they loaded the homepage. Also funny because now everyone has 100Mb+ internet, nobody noticed a few extra seconds on load.
There are lots of solutions to this if you actually need to serve a lot of bytes, but you can get stung even with very simple deployments. Not everyone wants to put Anubis or Cloudflare in front of their site.
Cloudflare offers a free plan that's fantastic. The free plan gives you effectively unlimited DNS/CDN traffic for a normal site, while the main practical cap is 100,000 Worker invocations per day, with 10 ms CPU per Worker request and a 100 MB request body limit. (If that means anything to you).
The next tier up from free is $25/month or $240 per year.
Does the free tier support geo-blocks? I'm experimenting with it -- running only a "hello, world" site thus far -- but haven't come across any offer of geo-block capability.
I run my personal website (and a bunch of other websites and services) off a somewhat more expensive but still reasonable VPS (I think 20€ at TransIP - it's so little that I forgot). Load is basically nil most of the time anyway. I think the bot problem is not so bad for personal websites.
Yeah, you just make sure your site is fast enough to handle more than 1 RPS.
But if you want them to actually stop, you can also just serve a little JavaScript page that sets a cookie and refreshes, to anyone who hasn't set the cookie. The DDOS attacker doesn't run JavaScript.
GitHub Pages, GitLab Pages, and Cloudflare Pages are all free. If you have a static site, you don't need to worry about lock-in, since your files are easily portable to a different provider.
I made this tool to try and make it easier for beginners to get something up on the public web for free: https://weejur.com
I host a couple WordPress blogs on Lithium Hosting. It's like $5/mo, fixed cost, and I've never had any issues. If I somehow hit a transfer usage cap, they would just shut my site down until the next month, but I've never had that happen.
My site's built in Go on a reasonably priced VPS and holds up well. I do geoblock traffic from China and a few other countries as well as IP ranges maintained by Spamhaus.
Static site. If you do go for a WCM of some sort you should make sure that if it has a calendar view of some sort it isn't unbounded but that it only goes as far back as your first post, and only goes as far forward as your latest. That's one of the many ways the poorly-implemented AI systems get stuck.
I have a low cost VPS (actually two in two different pre-AI datacenters) and it runs fine. I use nginx to serve the web pages, and the content is about 99% static content.
The vBulletin and PHPbb style forums have issues with slowdown (I haven’t had a forum since 2015; even back then those forums were overrun with spambots), but static content on a nginx site can be served lightning fast.
As someone who is moving my static sites over to GitHub pages: While they are free and work really nice, the problem is that GitHub frequently doesn’t deploy updates to the pages.
I have frequently have had to update a GitHub page, push the change, and then GitHub’s actions puke instead of deploying the change. The workaround is that I have a .txt file with a list of GitHub actions which failed, and when GitHub actions fails, I update that .txt file and push the updated site, which GitHub actions will hopefully successfully deploy.
GitHub pages are OK for pages which aren’t updated very frequently, but they are not OK for pages which update frequently.
A $5/month VPS is fine for mostly static content. I’ve got a site that has been pulling millions of viewers and sits at the top of google for its niche for a decade. It is now hammered with bot traffic constantly. But everything is cached so it’s fine.
All of my boxes are cheap VPS. Highly recommend people throw Cloudflare in front of their stuff. I switched all of my load balancers over to there and all of that bot crap went away. That combined with a proper UFW setup keeps the weather clear for me.
Until proven otherwise in your specific case, the better technology is just hosting directly on a VPS without Cloudflare. And if your 5$ VPS is maxed out, while serving any less than 10 requests per second, then you need to optimize your software before considering an upgrade.
Bandwidth is cheap at places that don't call themselves "cloud". You can even get unmetered. 10TB/month is usually enough, even with bots. If you're serving large files, put a simple bot test on the page before the files.
My single static webpage with no updates in 3 years is doing that, which is (one of the reasons) how I end up where that site (and many others in business and personally) is.
You're chasing a dream for a world that doesn't exist anymore.
I'm on a $4/mo droplet on Digital Ocean and my static site has been just fine. I'm using Caddy and it seems to handle the load like a champ. My site is very lightweight though, so ymmv.
Cloudfare has a very generous plan usually for hobby projects. The article did mention that cloudfare did not work for them, but they have recently introduced few features to block AI crawling as well.
Seems about right. I rotated my logs this morning. Most humans go to access.log and most bots go to botpoop.log. This is the line count:
2 access.log [1]
40 botpoop.log [2]
2 is really 1 since a human will grab the CSS file. Most bots do not bother with the style-sheet so that's a 40:1 bots to humans. I could cut that down by blocking data-centers but then I inadvertently block a lot of VPN's which I really don't need to do for a static compressed blog served from ram. The bots just get a TCP Reset but it's still fun to log and study them. The most interesting one I've seen recently is ReadYou which may be a reader but it appears to be much more, possibly acting as a cell phone distributed bot collecting data for a centralized site.
Is there some existing mechanism already that counts how often an ip only scrapes the page and not the css and then block those origin IPs if it occurs “too often”?
Unfortunately, the best practice is to make css cacheable so you need to keep long histories.
I thought about that but to your point CSS is cachable. In fact I made mine immutable. No I just visually spot patterns and use that to study other facets of the agent, other client headers or lack thereof, supported protocol, accepted encoding and so on.
Maybe it is enough to include some css/js which is served without cache and is loaded after all user visible css/js is loaded. Make it small enough to not cause too much bandwidth for the server and legitimate clients. Then anyone who doesn’t hit that CSS file gets banned.
Possibly. I've noticed however that if people default to reader view they don't always pull the CSS. Seems to vary by client software. I'm weird enough to just stick with visual inspection and add characteristics when botters get too clever. Playing with the botters is almost as fun as Megabonk.
It's a sequence of thin lines, between using Chrome out of the box, to using something like Brave, to using a highly customized Zen Browser, to having ChatGPT look up a particular page for you, to using a small BeautifulSoup/scrapy script to scrape dozens of pages, to scraping the entire web. And at every point on this spectrum it's humans driving a "user agent" tool to make requests and process responses on their behalf.
I also see quite a bit of traffic from China and Singapore. I wonder if it's some scraping for AI training. It doesn't really bother me too much because traffic is still fairly low, but it skews all the analytics for me.
The experience of browsing the web has really suffered lately. The mandatory 3-4 second "verifying that you're a human" block from Cloudflare seem to show up on more and more websites. Seems like a questionable choice from Cloudflare to teach everyone to associate Cloudflare's logo with high latency...
A 3-4 second cloudflare wait seems good to me at this point. I had to just block any cloudflare requests because some pages that used it would just rev up a cpu core indefinitely and hang the browser.
blocking by geo yeah might work - but what happens when someone is traveling abroad ? they've to use a VPN to access your site ?
my take with all the bots - the web is gonna be a bunch of private walled gardens. with most sites set to no index. you will only discover them via referral from someone real.
Your comment kind of reminds me of the old slashdot spam-email-solution copypasta - the purpose of it is to highlight how so many sensible sounding ideas just can't work in practice.
# As a condition of accessing this website, you agree to abide by the following
# content signals:
# (a) If a Content-Signal = yes, you may collect content for the corresponding
# use.
# (b) If a Content-Signal = no, you may not collect content for the
# corresponding use.
# (c) If the website operator does not include a Content-Signal for a
# corresponding use, the website operator neither grants nor restricts
# permission via Content-Signal with respect to the corresponding use.
# The content signals and their meanings are:
# search: building a search index and providing search results (e.g., returning
# hyperlinks and short excerpts from your website's contents). Search does not
# include providing AI-generated search summaries.
# ai-input: inputting content into one or more AI models (e.g., retrieval
# augmented generation, grounding, or other real-time taking of content for
# generative AI search answers).
# ai-train: training or fine-tuning AI models.
# use: how AI systems may consume the content (immediate, reference, or full).
# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
# BEGIN Cloudflare Managed content
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /
Perhaps this could be construed as a license, e.g., permitting or prohibiting certain uses of the "content"
If, for example, the website operator had enforceable intellectual property rights in the "content", such as copyrights, then perhaps the operator could restrict access to the "content" under the threat of litigation to enforce those rights
Basic questions
1. Is the "content" protected by intellectual property rights, e.g., copyrights
2. Does the website operator have intellectual property rights in the "content", e.g., copyrights
3. Does the website operator have agreements with the rights holders, e.g., granting the operator authorization to restrict access to the "content"
It'll be the Chinese. The big Chinese cloud providers have data centres in Singapore for some reason. My Chinese mobile phone sends all it's telemetry back to Singapore, not mainland China.
at this point the web is mostly machines politely asking other machines for permission to read each other's content, and humans are the ones triggering the captchas.
I'm working on a web app right now, with the intention of it going to be 100% paywalled. It's 95% complete, but the remaining 5% is just implementing the paywall. In the meantime, the app is live and operational with a fully functional signup. I am deleting about 100 new bot signups per day right now, it is crazy out there.
I'm blocking the Amazon search crawler, anything coming from Googleusercontent, and limiting AI crawlers to search rather than allowing AI assistants. The residential proxy waves are something to behold, but Cloudflare does an okay job catching those in the AI labyrinth. Still, it's all a bit silly, and I can't imagine what large sites deal with when I'm tangoing with this much nonsense on a small tech blog.
Sigh .. same here. I don't write blog posts often (enough) but the ones I do write are from personal experiences and I take a lot of care with them. I look at my logs snd see bots everywhere, but now I just let them get on with it. AI scrapers are different though; they get to read my content which, just for them contains a smsttering of finest digital toxin. A pox on your datasets!
The annoying part is a large percentage of misbehaving bots (not obeying robots.txt for example) are via end user proxies across the world. However most of these aren't doing full-browser loop, so if you are behind cloudflare, you can do non-interactive challenge and that can help quite a bit.
Previously, I had done a fair amount of research into how Google's monopoly on web crawling further entrenches their monopoly in the search engine market. You can read more about this here, https://knuckleheads.club, there is a long report from ~2020 or so that explains how it worked at the time. The club is mothballed, I am doing other things with my life, and I'm happy to say that we played a very small role in the DOJ ordering Google to share their crawl data with qualified competitors (a work in progress, but it's progressing).
Chatbots have super charged this dynamic though, to the point that it is showing up in the robots.txt data. The last few weeks I've been having Claude rerun some old analysis of Common Crawl from back then, when I have spare usage and time. What I've found is that you can see pretty clearly the rise in people outright blocking AI chatbot related crawlers likely because of how aggressive they have become.
GPTBot is OpenAI, ClaudeBot is Anthropic, CCBot is Common Crawl, Google-Ext is a way for website owners to indicate they don't want their content to be used for AI, Bytespider is Bytedance, Bing and Google are the last two. Take these numbers with a truck of salt, haven't had time to verify them.
It's very clear that website owners do not like getting their content scraped and are indicating to GPTBot et al. that they are not welcome. It's a shame that CCBot is caught in the cross fire, but that's life. Bing and Google are doing just fine though, almost like having significant power in the search engine market gives you an advantage in other markets too. Who knew!
This is a static website running on Cloudflare infra. What on earth costs $90 per month? First optimize your infra before throwing up rules in front of your visitors. I have several websites on Cloudflare too and I don't even check how many million requests I get. Because it does not cost me anything.
> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.
Being self aware does not make it okey. Either you are okey with scraping (like me) or against it. Don't use it yourself and block your site at the same time.
These same people will be crying about how Cloudflare ruins the internet because they get these captchas.
I don't look at the logs of my personal site very often as it is a static site, but just went to check, and yeah, it's almost all ai crawlers. Note sure what is going on here, but I hope this isn't really anthropic:
I despise cloudflare. They've decided that my home IP address is bad, so I have to capchas for most websites. Sometimes on infinite loop and I never get to the website. I even reset my home IP address more than once, but it instantly continues. Especially if I use any VPN, even my work VPN.
Most people don’t know they’re acting as a residential proxy, lots of devices and apps and free tools install spyware which often includes a proxy script. It’s a really shady market.
> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer. That's usually the easiest way I spot a bot.
> And they were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch.
> But those 4,000 bots were just the warm-up.
I just hate this style of writing like you're on Twitter. Why does the above need to be 3 different paragraphs? A paragraph break indicates a separate thought but the author is still talking about the same data and still making their point. The sentence "But those 4,000 bots were just the warm-up." is effective when still the last line of a paragraph and it signals respect for your readers. I stopped reading after this because it's just a terrible reading experience.
Here's a correct version that doesn't read like the author left for a week to think about what the next sentence would be or having some sort of anxiety-induced mental pause:
> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer, which is usually the easiest way I spot a bot. They were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch. But those 4,000 bots were just the warm-up.
And you're totally right: I mostly post on X (nee Twitter) and I probably have ADHD or just a low attention span, so I prefer to read things broken up into paragraphs. But for a smarter audience like this, and that reads long-form blog posts, I should tighten it up.
Thank you for the suggestion. LMK any other edits and I'll be happy to tighten it up.
Glad to hear you’re willing to accept feedback. You maybe don’t have ADHD and sorry for continued advice but you shouldn’t assume you have undiagnosed conditions IMO as it allows you to defer your mistakes from the self. Even if you do have ADHD you can still correct and understand good article structure. I highlighted the last sentence of that paragraph because what is clear from the split sentence style is that you’re writing for impact, this lends well to 140 characters but falls apart in longer form writing but as I stated the sentence is still impactful as you’re saying “but wait…there’s more to this!” Which is very intriguing. I think if you’re comfortable writing that way and it helps you split your ideas and sentences up so each one is impactful, that’s a good thing. But consider that style as a draft and then you can go back and group up your impactful ideas into paragraphs very easily.
For other edits I think my qualm applies to other parts of the article but I found that specific paragraph the most impactful way to illustrate what I was talking about. I leave the rest to you as a challenge. Don’t lose sleep over it, there will be more writing in the future to apply it to.
As for the ADHD stuff and the urge to self diagnose consider something less severe but similar symptoms. Have you considered VAST? Here is a good HN comment briefly detailing it and mentioning a book (titled ADHD 2.0 I believe) that may be more in line. I am not a doctor of course and VAST is rather new.
Most AI detectors are dogshit, though. AI is trained on human writing and tries to imitate it, while these detectors basically flag certain writing styles as "AI-generated."
The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see.
A second side effect of a knee-jerk reaction to bots crawling websites is that if you try to fight all bots, you also end up hurting real users that use "bots". If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. That might or might not be what you expected, but it's worth taking into account.
And finally, something worth noting is that there are so many websites whose owners complain about bots, but the real problem is that the website is poorly built and should be improved anyway. Bot traffic is not necessarily bad.
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user.
No; in this case you are not a user, you are a bot user.
I don't disagree, but there is a sliding scale here.
For instance, I wanted to buy a piece of equipment the other day from a local company for a specific usecase. I wanted to find a specific price/weight/specs ratio, and asked an llm to loop through the 20 or so items, fetch their page and calculate and present some values for each. This then led to me going and buying the one I found. So the llm was mainly just an extension of me clicking into every page and making a spreadsheet myself.
However, if it were to continuously poll, or just scrape or something with no intention of buying, I would be no better than a bot.
No, there isn't, it's a bot. Justifying it with a strawman case study doesn't change it.
This is giving "why is this plane full? What in doing is important but what is everyone else doing here?"
I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. They clearly don’t mind so what’s the issue? You still used a bot.
> I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you.
The point is that not all bots are bad. Assuming as much / implementing policies to that effect won't block all bots but it _will_ block the portion of bots that represent users considering giving you money.
I have a series of bots that monitor various eCom sites to monitor prices over time for big-ticket items I am considering as well as staples/groceries and everything in between. I have this little scrape/ingest pipeline because there's no other way to obtain this data... not even an API that I can pay for access to.
One of the large appliance sellers that I have in the scrape queue has gotten _hyper_ aggressive with bot detection to the point where even my personal head-full chrome instance doesn't always get to load the page. Guess who I will never buy that ~$2000 appliance from.
If this is the case, they probably outright do not want you as a customer.
Significant sales (beyond for excess stock clearance/discontinued items) exist largely to generate some excitement from potential customers and to lure them in to taking a look at the site/store and to hopefully get them to buy some extra stuff that's less discounted while they're there.
The actual on-sale item is potentially losing money for the business or at break-even.
------
A person who's running a series of bots like this to track the exact lowest price on the internet for items over time and to always only buy at the lowest price it's ever been sold for....is probably the least likely kind of customer to buy anything extra that's actually profitable for the business.
> If this is the case, they probably outright do not want you as a customer.
Almost certainly! No eComm site wants the cautious shopper that's done their comparisons and knows price trends; they want the impulsive people with a line of credit. I doubt that my local grocery store is happy that I index all of my receipts so I know what I buy, how often and when the prices change so I know when a discount on something I buy every once in a while is actually a discount.
When I built this a few years ago, I had to write a lot of code to make all this work.
Now that's a simple "hey $agent, check the price of https://some-site.com/appliances/that-new-fridge.html and https://other-site.com/shop/appliances/that-new-fridge.aspx daily and notify me when it's on sale or has a promotion"
I would expect that more and more people will discover that their LLM of choice can do this, that their LLM of choice can help them comparison shop and do analysis of when - historically - is the best time to buy that new TV or whatever (yes, pricing on this stuff is seasonal!).
Maybe _most_ people will never do this, maybe I'm wrong. But if i'm not, blocking all bots "because bots are bad, mmmkay" is how you loose a sale.
Ignoring the specific price data example, more and more and more people _are_ browsing the web through an LLM. Assuming that continues, blocking bots is going to have the same effect on your traffic as doing everything in the "what not to do" section of the google SEO guide book.
The tradeoff here is a classic false-positive vs false-negative issue. If the cost of the bad-bot false positive (which blocks your bot out) is lower than the cost of the bad-bot false negatives (which allow bad bots in), then it is still a good tradeoff, if a suboptimal situation.
The fundamental issue here is the way we use the internet is changing… and what we are seeing is some friction as that works its way through.
I have a business where I clearly don’t want any bots. A retailer, maybe they do.
you know what a browser is called by the web site? check the header that tells the version.
USER AGENT
not user, an agent on behalf of the user.
the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT.
for a glimpse at one possible futureverse, check out what the home assistant community is up to, how they assemble then update the ambient information displays on their walls. their USER AGENTS are doing what most HN-style "hackers" dreamed of reading scifi as kids.
the alternative is all your in home information owned by corpos when the info should be from user agents not corpo feeds. if you want to vote this idea down, you might be a corpo. :-)
Yeah it's sort of weird that people are so used to using their computer to interact with other computers across the internet that the whole thing has become invisible to them: They completely forget that they're using a computer at all. Along come "bots" and suddenly everyone remembers that the whole thing involves computers. Driving the computer by hand for hours was never efficient, but is marginally entertaining I guess.
This reminds me of all the confused comments that pop up whenever someone calls plain HTML with forms and links a "REST API." "But APIs are for programs! People visiting websites aren't programs."
https://news.ycombinator.com/item?id=32141027
I fancy the approach myself... but, much like the grand ideas of OSI in networking, Tim Burners Lee's idea of the Web was never what was going to be used and a lot of the original terms/concepts just plain don't make sense for the Web that exists. It was, originally, a true document system, nothing more. It wasn't until many changes from the original idea that it started to be what people wanted & actually use today. Nor was it even part of Tim Burners-Lee's original concept that 99% of Web traffic would be something other than users viewing & editing HyperText documents.
The User Agent would get a HyperText document from a Web server, render it, and show it to the user. You could parse HTML, display images, and follow links - with the assumption the user would want to edit them like any other document. There was no encryption. There was no scripting for dynamic content. Styling was done via semantic markup. There wasn't even any POST when the User Agent term was initially applied. Everything was supposed to be like one big distributed document store instead of centralized hosting on few servers for many client devices.
All that is to say: Regardless what I think sounds nice myself, if the web was just the original vision & definitions instead of everything it changed and grew to become we wouldn't even be using it to have this conversation.
Interestingly, people seem glad to have telephony companies filter bots even though the original idea of the telephone did not call out robocallers as invalid clients. Probably because, unlike the modern Web, the average person gets a chance to see what it means to allow anything inbound even though the original concept never considered robocallers.
I was on the internet before HTML and long before Mosaic...
I agree with one aspect of the narrative you're saying, his idea was held as being for «documents». To be sure, document was more abstract than document on a Mac.
Either way, I'm very sad it mostly turned into animated brochures instead.
But the reason styles were separate, the reason everyone talked about semantics, was always about the universal knowledge DAG and enabling machines to help us navigate and retrieve it. It's why so many of the early agents referenced navigation (Spyglass, Navigator) or the interconnectedness of things (Mosaic).
So, ok, big document “store”, but that's a lifeless noun. The information you are looking for, the dots connected for you surfacing what you didn't know, that's a living growing thing, an agentic retrieval savvy knowledge engine, distributed for curation and resilience.
The librarian was always to be a machine or many, on our behalf.
> for a glimpse at one possible futureverse, check out what the home assistant community is up to, how they assemble then update the ambient information displays on their walls. their USER AGENTS are doing what most HN-style "hackers" dreamed of reading scifi as kids.
I would love some pointers to read up on this, what are some examples of this? Genuinely, not being a troll or ironic
> I would love some pointers to read up on this, what are some examples of this? Genuinely, not being a troll or ironic
There's a "look what I built!" thread every few days on the /r/homeassistant sub and the home assistant forums, too.
The data sources vary but it's usually a mix of devices that the user has locally and some remote data sources like weather/public transit schedules APIs or RSS and in some cases also utility rate info all integrated into a SPOG that is then displayed on some ambient screen (usually a hacked tablet or other cheap ARM device)
It's an interesting example. People did very similar things during the "desktop shell modification" scene in the early 2000s, by far the most common thing people modified their desktop shell to do was display a bunch of charts and graphs of semi-random information, usually about their local computer.
It was useless in 99.9% of the cases. You don't need or want to have constant access to how much free ram your computer currently has, as if that even meant anything.
So, after a few steps, that leads to the question "what is useful information? How do you define that?"
A basic attempt at it might be: useful information is something that causes you to make or change a previous decision. Take an action you wouldn't have otherwise done.
If you're living in the american southwest, and it's between the months of like, march and, dunno, septemeber, you probably don't need to know the current outside temperature to decide if you're wearing shorts or a jacket when you get up in the morning (you're wearing shorts because it's always hot)
Theoretically there could be some kind of "extreme" weather, aka rain, that causes you to do something different, bring an umbrella, but it's pretty difficult to not notice it currently raining, even if you're inside a house.
So what other information do you want? There's all sorts of stuff that's considered "news", but I'd argue that at least 98% of that is basically just entertainment. We'd need some kind of article or perhaps even a book to hammer out all the nuances involved, but seeing a headline in the morning that trump has caved in to iran again or killed some more random people on a boat, while sort of important to know, doesn't really change what you're going to be doing that day.
Knowing ICE is trying to do some domestic terrorism near where you might be could be useful information, but there's not a real good way to track that.
So yeah, what kind of information is actually useful to have in the morning? Stuff like calendar appointments would count, knowing you have things scheduled at specific moments or tasks to do, go grocery shopping, etc, todo lists in otherwords, but what else? That's basically all self-generated stuff by you about you, you don't really need any outside data sources.
> Stuff like calendar appointments would count, knowing you have things scheduled at specific moments or tasks to do, go grocery shopping, etc, todo lists in otherword
So ... most people.
> That's basically all self-generated stuff by you about you, you don't really need any outside data sources.
Other than the ones you just enumerated that most people are likely to use, I need to know - using external sources:
- When to expect the garbage truck (do I put the recycle out tonight, too?)
- Is it a school holiday?
- When it's cheapest for me to charge the car up; if there are any "grid stress, please turn non-essentials off or we're getting rolling blackouts" alerts.
- Same goes for nearby fires or other issues that are going to change the "when do you open/close the windows and switch to AC" math.
- If it's likely to rain in the next few days so I change what plants get watered / when
- Packages that I am likely to get in the post today
- Weather (for display, not just for use in the automations mentioned above)
- Where my power provider is currently getting its power from (so I can calculate estimated CO2 emissions, if any)
And that's about 1/2, probably. Can't be bothered to go spend 20 min pouring through my HA config to build the full list.
I also stream the npr newscast as an MP3 as part of the "good morning, while you wait for coffee..." automations, if that counts for anything.
Can you recommend a specific home assistant community to check out?
r/homeassistant
> you know what a browser is called by the web site? check the header that tells the version.
You mean that key whose only legitimate values also call every browser Mozilla? Arguing from the dictionary is so tedious. I'm going to scrape your site, but the idea that you have some obligation to like it because of the definition of the word "agent" is profoundly stupid.
A lot of people live off ads. I block ads. People who live off ads aren't obligated to like it - and are free to take any means to block it without moral hazard.
This website refers to browsers as browsers. Just because a header in the underlying protocol is named a certain way, that doesn't necessarily reflect what the humans intend.
https://www.ycombinator.com/legal
I downvoted it because you don’t use capital letters and still feel inclined to make people read this rant. Corpo is a really stupid slur and reading this rant was a waste of my time.
So instead of adding of addressing his point about User Agents, you decide its better to make a low level bullshit comment about the very last sentence to stand up for... checks notes... Corporations and dismiss everything else. Top notch quality content that for sure added to the conversation.
> not user, an agent on behalf of the user.
Identified by the user agent header. Which most bots fake or leave out to increase the chance of getting where they are not wanted.
Your bot is a good bot? Great, let us know when you've dealt with all the bad bots and we'll open the doors to the remaining (good) bots again.
"Bot" access on behalf of users should be fine, even preferred. The missing piece is some micro transaction layer and some sort of attestation somewhere in the layer that the person driving the bot is not a bad actor.
Equating bot with bad actor in 2026 is Luddite behavior. Driving people to your site so you can serve them adspam or just make whatever operation they want to do 10x more difficult is the same.
MCP style APIs should eat the web. This doesn't mean the "open" web goes away. > 99% of the time I don't care for a domains particular FE at all. It's a complete waste not only of time, but resources and bloat. Pushing the contract into the agent should become good UX. Making things harder for good faith users should never be the goal.
The best way to read the information on the internet today is via a LLM.
Just like you don't process raw data in a data warehouse or datalake by looking at tables, you use a SQL or BI tool, to process all the information on the internet you need to tool to digest it for you. For many, today that tool is a LLM chat interface or agent.
The worst way to read the information on the internet today is via a LLM.
This is such a grim thing to read.
I disagree. I personally consider these my biggest problems with the Web:
The LLMs are very good at reading from multiple sources, parsing, and presenting only the data in a consistent format. There are many examples of this, but if you want a good one to try for yourself:Google "How to make ham fried rice"; you'll get 10,000 articles, most pretty good recipes. But they're all different; most of them are just bait for ads. And most of them, the 10-line recipe is hidden between 50 useless paragraphs about how serving food is life's most important goal.
Now, ask an LLM to search for it, find the best combination, and list only the recipes. You get a perfect, 10-line recipe that doesn't waste your time.
I personally prefer to read a couple of recipes to understand the context, what people have tried, what they thought of it. Only when I have a good general idea of the recipe I choose the best one I found based on difficulty and the ingredients I have. I then do the plate, often not following entirely the recipe I chose, and after that and eating it I write down the name of the plate, the recipe link, the date, and a comment about what I did and how it went. Cooking is a lot more than executing a list of steps like a robot.
LLMs are not unbiased. Nothing is unbiased.
Yeah I had to laugh at that. LLM's solve commercial bias? Coming from seemingly multiple the most valuable companies of all time?
And if you're lucky, it won't include rocks as an ingredient.
They're an important part of your diet if you happen to have a gizzard.
I've also been known to include some ground up rocks in my meals. I'm pretty picky though about them, I mostly just want a couple specific varieties.
Why even use a LLM, just buy a cookbook and you can even waste less time and know the information is valid. And again if your logic is consistent, google routes your search to an LLM automatically and renders the reply - that will save as much time and do the same thing since LLMs are so good at parsing and presenting information.
So why should I pick an LLM over google's inbuilt LLM or an actual cookbook on the desk? What wasted time is the LLM saving?
Well agreed, but it's not new, the web has been actively made user hostile by every website owner and their "273 partners".
Maybe, but it's reality. Not too long ago the rule would be "if it doesn't show up in google, it doesn't exist". Today agents+LLMs are better browsers than Chrome/Safari/Firefox/... If a webpage does not show up in an LLM it may as well not exist.
How is that grim? It's the dream of the Semantic Web coming true, just by different means than planed.
Because the economic, social, and climate impacts are at best uncertain and at worst devastating to the majority of human beings on the planet. It's always surprising to me that people don't intuitively separate the usefulness of AI from its risks. It feels like everyone has to be a doomer or a booster.
Why do you feel that ?
Because AI cannot be trusted as a reliable source of information. Information is more trustworthy when it comes from from primary sources on the open web.
Why not ? Can google be trusted ? Or facebook ? How do people have been accessing the internet for the last decade you reckon and how's this any different ?
Google (as it originally) existed wasn't a source of information, it was an *index* of it. You were trusting google as a source for {search_query}, you were trusting the links it gave as a source (based on your own evaluation). LLMs are fundamentally different, because you are trusting the software to actually generate the information in a truthful way.
If you use google (sans AI), you're putting some trust in their page ranking algorithm. If you use it with AI, you're trusting the same algorithm (since that's how the model gets it's sources), but then you're trusting the model to evaluate the sources for credibility and extract the information you actually want.
I routinely run into Claude Opus and Fable making basic mistakes like misunderstanding a simple negation, which would be on top of whatever reliability issues there are with a source. I think that means it is functionally very different.
> Can google be trusted ? Or facebook ?
No. Neither can be trusted as far as you can throw them. They're both incredibly invasive data brokers. Their customer facing products are just vehicles to show ads and Hoover up more PII and behavioral data on everyone.
Well then ask the LLM to go find and pull its information from primary sources. Don’t ever rely on its own training data.
100%
I think it's time for people to build a local database of every site they've ever visited, and then they can give their LLMs access to that. I might be willing to pass this data off to a company to store for me. Companies already store all my emails and money--why not trust them with this too?
Like, all the comments of this post would go into my personal database simply because I loaded the page, and it would help me find old information I've read, and could also inform LLMs I use.
This should be built into browsers.
I assume you are unfamiliar with the concept of user agents? Otherwise your browser would count as a bot user no?
And if not, what if it was a custom browser and not Chrome/Firefox/Edge?
There is bot traffic initiated by a human and bot traffic not initiated by a human. I would want to serve the first but not the second if it impacted my cost/performance at all.
Isn't all bot traffic ultimately initiated by a human? Someone plugged the computer in and gave it instructions. It may result in one http request or billions of them, but the human is still the initiator.
Yeah but you know what I mean -- A person specifically interacting with my brand vs anthropic hitting all sites 100000 times a day
What if I’m a web alerts company? I crawl your content but my clients are all end users who actually see your content at your site.
Fetch my RSS then.
The vast majority of news sites don't have an RSS. They don't even have a robots.txt.
Cloudflare and other WAF blocking is increasingly an issue with RSS feeds, especially if you use a centralized reader.
Then you are a uninvited bot that crawls my content in order to send alarms to people to tell them to visit my site?
This is a "what if it's for your own good?" argument. What if I break into your house to clean your toilets? Although amusingly, if you're a "web alerts company" that I didn't contract and sends out alerts in batches on your own schedule, you will probably send a ton of your customers to my site at the same time and slashdot me off of the web entirely.
No, I won't send a ton, I'll send a few dozens to a few hundreds at most because not everyone is interested in the same things. And they'll visit at their own timeline. If your site can't service a few dozen requests simultaneously then you probably aren't a news site in the first place so the whole argument is moot.
Scenario one: you use software to connect to their server and download a webpage. You are a user.
Scenario two: you use software to connect to their server and download a webpage. You are a "bot".
Make it make sense
You’re missing the part about the human who interacts with the webpage.
because they maintain the websites for social status, if the user never sees the specific website or knows they used it, you can't gain any social status
> they maintain the websites for social status
Are you implying that's a bad thing? Doing things for social status has been an integral part of society for millennia. It's a legitimate motivation that can benefit both the recipient of the status and the rest of society. In this specific case, if you assume that the author makes content that is useful to you only because of the social status reward, taking away that reward means the author will stop making content that is useful to you.
One more reason for the bot difference is the llm bot users are doing almost all the traffic, and it feels wasteful, painful, and there are reports of 99.x% being llm bots just hitting webpages over and over pointlessly.
This is a change. People put up new caching layers, and it urks people running a website in a single small machine. Yes, this could always happen with slashdotting but it's different now.
I am denied by cloudflare CONSTANTLY on one system.
I have an old os (macos 10.11), running the highest firefox esr I can run, and I get denied by cloudflare.
But not always immediately - I get to enable javascript/cookies sometimes just to be denied.
they are not the folks we want gatekeeping the internet, they are opportunists increasing their OKRs
And here's the rub -- the bots are "running" the latest "MacOS" and have no problem accessing the site.
> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.
I think you've hit the nail on the head here. I think that's a big reason why people want to ban bots.
The social contract of accepting scraping for visibility was always tenuous and is now fully dead. But it’s shocking the number of people who are essentially victim blaming here. You should spend YOUR time to optimize your free website so MY use case is not impacted. How is that not incredibly selfish on its face?
The social contract for putting a website on the public internet involves following standards and making your website accessible. You SHOULD make sure that your website is readable to people no matter what user agent they are using as long as it's standards compliant.
For example, making a website that only works in Edge and refuses to load for anything else would be a shitty practice. You are free to do it. There's no one stopping you, but you should expect people to bitch about it and they might think you're kind of a jerk or maybe just bad at making websites. It's the same thing when a site actively rejects traffic from curl, wget, or any other common utility that many people find useful.
The social contract for putting a website on the public internet was built on the assumption that browsers would be used to access those websites, and that most browsers would generally behave in a similar way (click on page, display page, pretty simple).
LLMs are not browsers, nor are they people, so I don't see why they should be part of this equation at all. They do not display the page for a real person to read, they merely ingest its contents and then spit out something completely different.
> The social contract for putting a website on the public internet was built on the assumption that browsers would be used to access those websites, and that most browsers would generally behave in a similar way (click on page, display page, pretty simple).
No, not browsers, user agents (some of which would be browsers) and there was never an assumption of how a website would behave on the user's end, that's the job the user agent. All that HTML and CSS are only suggestions, but the power was always intended to be left to the user to decide if/how they wanted that data presented to them and it was always intended that the user be able to choose whatever tools they wanted to collect, process, and display content pulled down from the internet. That same principle is how we have ad-blockers. You are free to infest your website with ads, but as the person requesting the website I'm under zero obligation to display any part of that site I don't want.
> LLMs are not browsers, nor are they people, so I don't see why they should be part of this equation at all.
LLMs are just another tool used by people to collect and process the information available on websites. Maybe there is a distinction to be made between people using LLMs to get web content and corporations scraping websites to take training data, but even scraping has always been a common and expected practice. It's the current scale that is making things different.
Don't tell this person about translation plugins. They change everything! The end user never reads what the website actually served!
It's probably a pretty common misconception that websites should have total control over how their content is viewed, but it's strange to see that on a website like this. Part of the reason apps are so popular with companies is that it gives them control over how content is presented which is something webpages were never intended to give them.
Exactly and Cloudflare (there should be others out there too - open source options) has proposed that AI bots need to pay for access to our websites. If they and or others could pull off blocking AI access until it pays creators then AI is forced to pay as it should and always should've!
Overall systems and different business models need to be created that forces AI to pay it's fair share! Elon says there will be an abundance thanks to AI and we wont have to work -- ok how's that gonna work without different systems in place paying us?
I feel strongly about this topic and proposed some systems in this Substack post https://ryanspahn.substack.com/p/ai-to-pay-for-all-americans...
I'm getting a "browser not supported" by the Cloudflare check. So I guess the "job is well done", and the user is lost.
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user.
People trying to block bots end up keeping out all kinds of users. I get blocked frequently for using a regular browser, just with JS disabled. 99% of the time, I just close the browser tab and move on with my life.
Agreed! Cloudflare absolutely destroys user experience and honestly doesn't seem that effective in practice.
What's worked for me is I block any client that don't support brotli compression and http2. Seems to work well enough for stopping scrapers.
How do you monetise bot traffic?
You take your site off the public internet and paywall it off. Websites on the open internet should be accessible and ideally the goal should be to share something cool with the world, not just to make yourself rich.
This. The Internet was a better place when people hosted things they wanted to share at home or on their ISP's servers, without the expectation of making money.
> You take your site off the public internet and paywall it off
I did that. I even printed and bound it in a bunch of dead trees and put it up for sale. The LLMs still stole it.
Paywalling off the site solves the problem of monetizing bot traffic, any bot crawling your pages paid you to be there, but it can't fix the plagiarism/copyright infringement problem
Well... They paid someone to be there, not necessarily you.
https://claw-guard.org/adnet
> Revenue settles in $CLAW tokens
Yes, but no.
pa-ro-dy
https://blog.cloudflare.com/introducing-pay-per-crawl/
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user
You are mistaking yourself, well your bot, as his target audience.
You might as well say “If I want to send you my commercial email, and you block it, you hurt me, the email user.”.
While your point of being concerned about cloudflair becoming a global arbiter of who gets in and who does not (which may at times not just mean blocking bots, intentionally or through technical issues), the need to block the deluge of bot traffic is very real for many sites and that is one of the easy options for them to deal with that. There are other methods like directives in robots.txt and nofollow attributes on links, but so many bots simply ignore those that they are not really useful.
> Bot traffic is not necessarily bad.
Nor is it necessarily wanted. In fact, it often isn't. Unfortunately practically all bot runners seem to either assume that their traffic is the special good kind or not care either way.
> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.
I WANT! I WANT!! I WANT!!!
Well, that site runner wants you to access the site as a human, if at all, not via bots. Sorry to be the one to break it to you, but what you want isn't always the most important factor for the rest of us.
> but the real problem is that the website is poorly built and should be improved anyway
Firstly: just no.
Secondly: if you and your bot don't like our badly built sites, feel free to go get your information from those that you consider to be better built. Problem solved.
Jokes on them, the second I see that “Verifying you are human…” redirect I leave and never return.
> If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse.
If the the processing is subject to GDPR (e.g. if controller is in EU) then you do have recourse. You can complain to DPA or sue the company. The company is ultimately responsible for the decision to block you, at least in cases where you personally tried to access the site.
> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.
Substitute the word “website” for book, or training course, or documentary or published paper, or patent or one of hundreds of examples.
Now you see the problem.
And if you believe that blocking bots is bad, then by all means provide a better alternative that doesn't increase costs for sysadmins. If you believe that the website should just be improved, then by all means feel free to provide instructions on what should be improved and exactly how so that we don't ever have to block bots anymore. I'm sure all the sysadmins having to deal with issues like this one will thank you
This is a whole lot of things you want and feel entitled to. Nobody has to cater to you - we can block whatever we want to block. And if our poorly designed sites bother you, that’s too bad for you. But your wants are not my problem. If you want someone to cater to you, pay them. You aren’t entitled to anything.
>That is not the open web that I would like to see.
Cloudflare is opt-in so I don't see that being an issue (yet).
Being opt-in doesn't negate the fact that it is closing the web.
I agree but it's a bit nuanced.
If you can still buy a domain, publish a site, and other people read it as usual, then the web is still open imo.
But I can see a lot of negative network effects if/when Cloudflare gets to control 60%+ of web traffic.
Internet walled gardens are all opt-in and still they've made the web a worse place.
why you blaming cloudflare that try to solve botting issue and not the Botters ???
you literally can turn off cloudflare and use your own solution
Sometimes the cure is worse than the disease.
True, but for whom?
The cure here isn't perfect, but much better than the disease of paying hundreds of dollars a month for scrapers which will never be beneficial.
Worse for the scrapers really isn't anyone's problem but the scrapers'.
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.
The article you're replying to describes in explicit detail how the bots and their operators have directly caused and continue to knowingly cause real harm to the author and others in similar positions, both financial costs and administrative/maintenance burdens that would not have otherwise been required.
You then respond "but if you block the bots then I won't be able to use the bots, and that harms me because I might have to read your web site myself..."
Are you serious?
> That might or might not be what you expected, but it's worth taking into account.
I would wager that for almost everyone who is blocking bots after getting functionally DDoSed by them this is absolutely an expected and desired outcome.
> And finally, something worth noting is that there are so many websites whose owners complain about bots, but the real problem is that the website is poorly built and should be improved anyway.
Both can be true. If you operate a git repository with a public-facing web interface for example there are going to be a lot of possible operations that are inherently expensive but also incredibly rarely used by normal users so it doesn't really matter, but the bots now ignore your robots.txt and are programmed to go after every link they can find, so they trigger every single possible expensive operation more times in a night than your actual users ever have in the history of the site while dividing requests across so many different IP addresses that rate limiting becomes impossible at the individual scale. These days they're even feeding the discovered URLs back in to their models to have them invent new possible URLs and trying those in hopes of finding content never publicly linked. They will send you thousands of requests for URLs that they literally made up.
Sometimes the site is in fact badly coded and operations that should be simple have higher costs due to bad design but you don't have to look very far to find situations where legitimately high-cost resources are exposed to the public because they're expected to be used in a non-abusive way. We should always be standing up against abuse of public resources, unless we want to lose them altogether.
> Bot traffic is not necessarily bad.
You are right, but whether it's good or bad more or less comes down to a cost/benefit analysis. As we've already covered infinite times, these bots being used to train LLMs cause significant real costs to the operators of these sites. What benefits do they offer in return? We know the clickthrough rates are terrible, so what other reasons would site operators have to make those real costs worth it? So someone can get a response back from an algorithm that confidently misinterprets or even entirely misreports what the data actually meant?
Even the most die-hard "information wants to be free" types who absolutely want their datasets trained on would probably prefer that the bots accessed the data directly via an API or downloaded a database dump rather than spidering and scraping a web interface intended for humans.
> the real problem is that the website is poorly built
This is almost always the problem.
This is currently the problem that GitHub is having too. If they had remained with their crusty old rails architecture, very little of this mess [0] would be occurring right now. They would have been able to focus all their engineering talent on scaling the product rather than inventing elaborate client side state synchronization mechanisms.
[0]: https://www.githubstatus.com/incidents/qcvjkzcs7j74
We tried to deploy private cloud github enterprise node back in 2018. It was pure garbage without CI integration. I am happy microsoft took harder long term decision!
Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software.
[1] https://anubis.techaro.lol/
It doesn't detect "real browsers". It simply adds some friction and is niche enough that AI scrapers have not bothered to bypass it yet.
Yeah this is one of those things that hurts some real users and AI scrapers have figured out how to bypass. Not worth it IMO.
How do you figure it hurts real users? The amount of compute/energy used on the proof of work is pretty minimal. You're using more when you watch a YouTube video or browse a JS-heavy web app.
Of course a sophisticated scraper can "figure out" how to bypass. It isn't trying to be foolproof, it's adding an extra cost to deter massive amounts of bot traffic.
I put it in front of my hobby project because I can't afford to serve hundreds of thousands of bot requests from residential proxies all across the world, and I didn't want to route all of my traffic through a third party company like Cloudflare.
I've been happy with Anubis.
1. It makes the web more annoying and time wasting.
2. I'm a real user and I have to enable JavaScript and run arbitrary programs from untrusted websites to access them. One browser that I use doesn't even support JavaScript (w3m). Fortunately it is unaffected because it doesn't have the string "Mozilla" in its UA and Anubis lets every user without that pass to not hurt legitimate bots like search engine crawlers. As said Anubis is trivially bypassed. The only reason it isn't bypassed is because it's simply too irrelevant to the AI companies.
I empathize, I hate the bloated JS-heavy web and avoid it when I can. I just don't see a better solution. The options for me are: sacrifice some legit users by using Anubis, or sacrifice all users by shutting down my service because I can't afford to serve the bot traffic.
There are false positives. Every time I've used a tool like this, I get reports from my users about it blocking their access. It's especially bad for users who use privacy oriented browsers like Brave or use VPNs.
For a personal site where you don't care about 0.1% of your users getting blocked, it's probably fine. For a business, blocking some of your paying customers isn't acceptable.
> For a business, blocking some of your paying customers isn't acceptable.
That depends on your business and the costs you're incurring from the bots. It might be justifiable to lose 0.1% of your customers if it means saving server costs from serving useless bot traffic.
I suppose it would depend, yeah. I find having paying customers being blocked to be completely unacceptable. Not only is a bad customer experience, but they tend to be very vocal about it. Posting online, leaving bad reviews, etc.
There is enough visceral hatred for the Anubis branding, I am surprised an AI skill for bypassing it hasn't been broadcast yet.
Most of the friction is just JS overhead for the computations, a compiled solver is like 1000x faster. If Anubis ever gets popular enough that scrapers care, it would be trivial to defeat. And last I checked you could bypass it by just modifying the user agent
I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.
It's actually a genius idea. If you are someone who the professional presentation of not having an anime girl on the loading page is required, then you can afford to fork over the cash to fund development.
Per personal communication, this is precisely why the developer did that.
I guess the disconnect here is a bunch of HN'ers believing professional companies and websites want to attach their branding to a sexualized anime character and that they are willing to pay to remove it.
Which one then wonders why they would install it in the first place.
Sexualized? It’s just a cartoon/anime character holding a magnifying glass
> is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.
Most people either don’t know or don’t care about “those subcultures”. I bet a lot of older people think it’s a cartoon figure of Betty Boop (nurse) and miss the furry bit since it appears and disappears quickly. Most people also don’t have a brand.
So it seems like you’re describing a concern that only affects a tiny fraction of people:
- Not interested in paying for custom branding, so obviously not a corporation or influencer
- Dislikes cartoons
- Aware of, and hostile towards, “furry” subculture
That has to be an exceedingly small fraction of potential users of Anubis, and given how much businesses and branders will pay to custom-brand something, I’d counsel them to stay the course. Sure, a few never-payers will never pay, but they wouldn’t have anyways, so they can cope with Nurse Betty or look elsewhere for a competing free product.
If you think about this in physical market square terms — in other words, a bazaar — it seems horrendously rude to complain about a shop logo sticker on a free product handed out to anyone that walks up and asks for it. If you want it white-labeled so you can write your own name/logo on it, you pay for the privilege of displacing their name with yours. But you don’t stand there and loudly complain that their shop mascot has dog ears while holding a freebie bag of product, without losing the respect of everyone who hears you doing so.
If you have a brand or personal image that you are investing in, you can surely afford to invest in paying for a branded version of Anubis.
I'd argue its rather functioning exactly as intended with regards to obtaining paid users.
Also this view seems a bit elderly. For most towards the end of the millennial curve and younger, anime is no longer subculture, its just general culture at this point. Although I would agree it isnt necessarily what you'd want for every platform and web presence.
I think that's partly why they do it? If you care about that, you should pay?
Or just have an LLM write the same thing for next to nothing...
Pay to change logo??? Just fork Anubis and remove one div... It's not SaaS first software, you're expected to deploy it on your web server.
I wish my rent would stop going up year over year.
They're not stopping you. They're asking you politely not to.
How hard is it to maintain a fork that changes nothing except the logo?
If you can't be bothered to maintain a trivial fork, then why should the author of anubis be bothered to serve your branding needs? It's not like you have a service contract or anything do you?
You just discovered their business model congrats.
haproxy-protection is an alternative.
It's MIT licensed, you are free to do whatever you want to it, including removing the logo. They're basically just saying "we'd prefer you didn't do this, but we're not preventing you from doing it."
It's open source, just ask your AI agent to change the logo.
Just keep in mind that changing the logo is a great way to move yourself down the priority list for bug reports and support.
Someone willing to take the 2 minutes to ask Claude to change the logo can probably also ask it to fix any bugs they find, or add any new features they might want.
that's really smart of them. Just pay to use the service you need.
you really revved the weebs with this one
Nah, I don't especially like the logo myself when I come across those sites, but I do like that "brand" people who want to make money off it and not pay dislike it even more.
You wish they'd give your preferable imagery for free and not make you feel bad for using their free software for personal gain. Your subculture is irrelevant and not special.
If you 'don't mesh' with that then you are not worth protecting anyway :)
I hope I’m missing the sarcasm here…
Anubis sucks because CPU is cheap for scrapers and hard for humans.
How is it hard for humans?
Because of this wasteful crap the internet is so slow nowadays...
Try opening gcc bug tracker on your phone: https://gcc.gnu.org/bugzilla/
This is like getting angry at cookie banners instead of all the companies tracking and selling your data.
You're complaining about the symptom (needing to have these checks) not the cause (if they don't, 99% of their traffic will be bots, the site will slow to a crawl and be unusable anyway).
In any case, I saw the dumb anime girl for about 2s then the site loaded. Not a big deal.
The Anubis challenge took ~5 seconds.
What do you propose instead?
The GCC bug tracker uses the meta-refresh challenge, which does not require JavaScript. Due to the fact that the server makes sure the client has waited at least 75% as long as it should, the HTML has to add one second to the meta-refresh wait. Patches welcome. Meta refresh granularity is in single digit seconds.
I'm assuming a bot running a headless browser instance can still get past it?
It's still valuable to raise the cost of scraping of course. I don't think anything can really stop a determined scraper from impersonating a human. I wonder though if a system similar to Anubis but mining some crypto would make bots _welcome_ - since they're paying for their traffic.
People tried this in 2013 or so, there's no point to it. Doing proof of work in javascript in a browser is so crushingly, pointlessly slow that there's no value at all. Some browsers also intentionally detect attempts to do proof of work and attempt to block it entirely.
> Some browsers also intentionally detect attempts to do proof of work and attempt to block it entirely.
Then they'd be blocking themselves from the website.
https://blog.mozilla.org/en/firefox/block-cryptominers-with-...
make your only legit users mine fake crypto to access your site, only costs them 5% battery on an android device
They think scraping = money so you can just ask them to scrape (copy paste websites into a text area) and it will feel like payment.
Which is trivially bypassed by an actual implementation of the proof of work in non-javascript, rendering it absolutely useless. The website is approximately 3800x times slower than native code, and hundreds of thousands of times slower than the CUDA kernel claude wrote. The "proof of work" is just non existent at that point, they're solved in milliseconds for what would take the browser version 10 minutes or more, it's security by obscurity being dressed up as something more.
Claude even made a nice little API server for it after implementing midstate compression, AVX multi way hashing, and a CUDA kernel. This doesn't stop the literal LLM it's trying to block from solving the challenges, it's really annoying that everybody is using it and claiming that it's something that's usable in the real world as a result of it using proof of work. It's obscure, and obscure is fine so long as nobody is pretending that it is secure.It is not absolutely useless, empirically, which you'd discover if you had a website getting hammered by bots and experimented with anubis as a countermeasure.
While dedicated scrapers/attackers could work around it, and they could do so much more efficiently than the client-side js, almost none of them do. Unless you like paying additional hosting resource fees to serve bots, it's a worthwhile option, and is less annoying to typical human visitors than cloudflare's interactive captcha/challenge which is what most people use.
The main author is aware that the algorithm is far from ideal for this purpose. See https://news.ycombinator.com/item?id=48869064 . If more bots start to answer the primitive challenge anubis uses now, that'll hasten implementation of a different algorithm.
Don't let the perfect be the enemy of the good enough. For now, the algorithm or challenge scheme almost doesn't matter. Since it's much smaller-scale than cloudflare's challenges, that's probably why very few scrapers and botnets bother to solve anubis's trivial sha2 pow.
Targeted attacks may not be repelled at all. That's not the point.
It does not even require the PoW thing Anubis does. I've setup a simple logic that just:
Checks for existence of a specific static cookie, if it does not exist, output a small page that sets the cookie via JS and reloads. Sadly this kills Noscript, but it would be possible to add a form in <noscript> that when submitted sets the cookie serverside.
Is this trivial to bypass? Yes. It still keeps out 95% of unwanted bots. Reality is most do not target you specifically they just want to mass-scrape with low effort. Running headless browsers is way more expensive for their op
I've extended this with a FCRDNS checked exclusion for Googlebot.
Another quite effective measure I figured out was checking the existence of Sec-Fetch-Dest header if the User-Agent claims to be a modern browser. If you don't want to close down too much.
Also, I only apply these rules to routes that are not cheap and cached.
That's not far from what anubis does for clients that are determined to have light souls. It doesn't always send a PoW challenge.
For a webapp that sets a long-lived cookie, that cookie could be used to bypass anubis completely, or lower the weight in anubis so that it doesn't send its pow challenge unless there are major red flags. If bots start to abuse that exception, it can be removed.
But the people you're defending against don't do that.
They also don't load CSS but for some reason the security theater PoW won the mindshare.
I once discover you can put escaped XML or json in css content. The purpose was to have static data sets that work cross domain. No headers to configure no letting strangers run all you can eat malicious js on your site.
Security theater is what gets the economic rewards.
Like any lock, it's mainly to deter less determined adversaries (which account for the vast majority)
The point of PoW access is not that its hard to bypass, it's that you cannot bypass it at scale.
>it's that you cannot bypass it at scale.
Define "scale". For any reasonable wait that you're willing to impose on your users, any PoW scheme heavily favors attackers. They have unlimited time and can be scraping even while they're asleep. Your visitors on the other hand don't have that luxury. You might argue that's not the point and it's only to stop dumb scrapers that are effectively ddosing your site, but if it's just dumb scrapers, you could've stopped them less onerous measures like tls or javascript fingerprinting.
If algorithm used is static and GPU-friendly, then what stops bypass at scale?
It's more or less designed for it, it's SHA256 with a break in the middle for midstate compression to be effective, and the difficulty system is based on a misunderstanding of how bitcoin PoW works ("number of zeros" is never, ever a consideration in bitcoin, it's a match to a floating point target).
sha256(challenge + ascii(nonce)) means that the first compression round of the function can be cached and the second compression round is just the nonce plus the cache. This is the same trick used in Bitcoin mining and would have been avoidable by putting the nonce first, so immediately any non-naive code has to do half the proof of work as the vanilla solver.
how so, can you link to any sources?
The prompt used for Opus 4.8 was:
Then Then it was done more or less, it happily made a local server that supports solving the challenges given to it in bulk with priority based queue and can tolerate potentially tens of thousands of requests a second with no issue. The CPU time spent solving the challenges is less than the SSL setup for the connections. The GPU version does in excess of 20GH/s (but with high latency) though I didn't really test it, I'm not using this for anything but proving a point that the LLM itself can write the bypass tools and run them happily.In a thread last month about scrapers, the author mentioned working on a switch to hashx.[1]
In addition, nothing prevents anubis from sending a wasm solver instead of js, reducing the gap between a custom native solver and a js solver.
[1] https://news.ycombinator.com/item?id=48869064
I'm almost ready to ship the wasm feature in the next version of Anubis after the one that's about to come out.
The big blocker is that testing against dozens of googles chrome to ensure functionality on abandoned smart TV oses takes a nontrivial amount of time. As an example of the level of debugging and the like required: https://github.com/TecharoHQ/anubis/pull/1684/changes/67621f...
My office gets very warm when chromesweep runs.
This is something that is complicated enough that even though LLM tools can help, it's not a magic bullet. It's just complicated in general.
They should make it mine actual coins for the site owner.
The more bots try to access the site, the more profitable it will be!
> My normal bill for running this whole site is around $90 a month. During one bad spike month, it jumped about 500%.
This is D1 - which has very surprising costs. you may just want to drop D1 and move to a static site. There’s no reason your site should cost this much.
There's a lot of people who host their site at extremely expensive places and then do everything they can to minimise unneeded traffic - instead of just moving to a cheaper host. Vercel is another popular extremely expensive host.
I'm always flabbergasted when I see what people pay and how much effort they need to invest to keep their cloud websites from eating them alive.
My allegedly more complicated VPS stack needs an afternoon of attention every two years when a new Debian major release is necessary, and costs have been predictable for 15 years, no matter what happened traffic wise.
The predictability of costs is underrated too. You don't want your hosting solution to automatically scale up to $20,000. You want that if it's overloaded it's simply overloaded.
Thank you! I need to tighten up my KV compression, which is actually carrying a lot of D1's load otherwise. We also had some bad queries some months, as the pages and database grew, that were counting the wrong things (or extremely inefficiently) and those have since been fixed.
I just checked Cloudflare for SignalBloom (https://www.signalbloom.ai, which I own and operate).
Over the last 72 hours, Claude-searchbot [1] alone fetched ~205,000 pages. Sent exactly 1 referral. There is a lot of free financial data on the site, hoping for real users to benefit from it. It is hard to not feel a little cheated out that Claude gets to claim "Found it!" to its users without me getting no credits or compensation whatsoever.
[1] Exact user agent `Claude-SearchBot/1.0; +searchbot@anthropic.com)`
Proof: https://i.postimg.cc/Pqc3SS8T/Screenshot-2026-08-07-at-5-33-...
> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.
There's a difference between someone running a scraping tool occasionally and bots constantly and rapidly re-scraping the same site over and over again
The author's website is responsible for storing its own data. AI services currently treat the entire web as their storage and cache layer.
This is an important context: the site is more likely to be targeted by scrapers because it is a curated collection of scraped information.
Do the bots care? Seemingly very little intelligence in many of them. Could be as simple as the site has more pages, so more traffic.
Loot first, ask questions later.
A curated collection of people who give away money. Was ever sweeter honey ever found in a pot?
Similar to the dose making the poison - the thing that jumped out at me in this blog was the ratio of scraping to visits. Unless OP is scraping thousands of times a day I don't really think they're in the same class as the bots they are blocking.
OP here: I'm not scraping thousands of times per day! Usually just a few times per year.
But you could just be one of thousands of bots targeting the same sources you’re scraping.
Who scrapes the scrapemen?
Live by the scraper, die by the scraper
Something suspiciously absent from this article is addressing whether some of these are in fact human visitors, but humans issuing chatgpt or similar queries instead of going directly.
Is it really a bot if it's in response to a human asking for some aggregate information about charities, triggering a web search and then following the result links to get details for the human? Well, clearly yes it is, but it's a very different proposition from this article's implication that "they have no throttling on their scrapers"[1].
> Challenge 46 datacenter ASNs. Humans don't browse from AWS.
People who have workstations in the cloud do.
> The bots use 99% of the bill and I pay 100% of it.
Running a site this way is always a wallet-DDoS risk.
[1] though yes, by far most will be pure automation with no human in the loop. It's an assumption on my part, but feels like a safe one.
> Running a site this way
What do you mean "this way". What other way is there to run the site?
cloudflares lack of spending caps and and usage alerts is a seriously unnerving approach
> Something suspiciously absent from this article is addressing whether some of these are in fact human visitors, but humans issuing chatgpt or similar queries instead of going directly.
That's not absent from the article, it's right there in the section titled "The Claude ratio". ChatGPT, Claude, etc. use different user-agents for scraping vs user-initiated requests, and the author notes that user-initiated requests were an absolutely miniscule fraction of the total traffic.
Oh, that's what "Anthropic's search crawler, had requested 420,680 pages in one week. That same week, Claude sent me 12 human visitors" meant?
Yeah, googling it does seem like "Claude-User" is for user-initiated requests.
By "Claude sent me" I thought the author meant referer header in real browser requests showed that they came from.
I mean, that's what the section "Pages crawled per visitor referred" refers to, right?
I'm in a similar boat. Probably 99.999% is bots. I have nearly 1m unique "visitors" according to cloudflare and my real users are in the dozens a day. That being said, I love the open internet and am holding on to keeping as much open as I can.
I shutdown all my little informational hobby projects this year that I've tinkered with for decades
They all shifted from mostly paying for themselves (or being so cheap it didn't matter), to essentially producing zero income while resource usage leapt up in magnitudes
I couldn't justify the stress and hassle of making sites, that a few dozen people a day might find useful, into some complex hyperscalable obligations just to feed the bots
But what do these bots gain from this?
It's all OpenAI / Anthropic / Singapore/China based crawlers. I guess they gain data that I provide for free. I get very little crawling by Googlebot / Bing etc which barely even index my site, I only have a single page "indexed" by Google.
Have you ever use an AI chatbot? It's basically a democratized scraper.
Type something stupid, it "visits" 200 websites and regurgitates some random gibberish.
Possibly content that appears only briefly? I'm not sure otherwise. It seems wasteful.
The most recent info of every single fucking thing on the web. Everything. Data locusts.
Can someone help me understand the underlying motivation behind this?
It makes sense that some crawlers, in the style of Google, would want to index the entire internet. But what is the point of the same crawler re-fetching a page they already fetched an hour ago? Or possibly all this traffic is just independent entities, each trying to cache the internet? The scale of bot traffic makes this seem unlikely.
What's the motivation behind the same entity re-fetching a page it just fetched less than an hour ago?
They are poorly implemented by the "fuck you I got mine" crowd. They will get stuck doing things like trying to run through a calendar that could theoretically go back to the beginning of time and all the way to the end of time. And because that calendar might change, it gets scraped for every inquiry made to the poorly implemented AI system.
Similarly, I've never understood the economics behind the constant rescraping that is flooding the internet or really what's triggering it.
It can't all be agents reacting to user queries. It's confounding how much CPU and bandwidth is getting flushed down the drain.
The most recent data on the internet for advertising, intelligence, etc.
And a lot of bad scrapers.
I run scripts on my servers on an hourly basis to check which are the top 25 IPs visiting the server (aggregated by /24). If anyone in those top 25 IPs are from China, Vietnam, etc, or from Alibaba/Amazon/etc, the /24 gets blocked by iptables.
It's far from perfect, but it was a quick way to get rid of bots, while not completely blocking people from countries such as Vietnam.
However, on a Gitlab instance I manage (500 users), we have to restrict viewing of git logs and pretty much everything except issues. The bots were too aggressive. Chinese crawlers have access to a huge range of IPs and they often do only 10-20 requests per day, while generating in total over 50k requests per day. Our server load went from 99% down to 0.1% after that (and it's a fairly big server).
My solution was similar. Anything coming from the ASN of a major cloud provider gets a CAPTCHA with a little nuance to allow Google and Bing to index. That seems to do a pretty good job. Also, anything coming out of China or Singapore also gets a CAPTCHA as my site is not popular in those regions and many Chinese bots seem to show up as a Chinese mobile provider. So far, the bots have never attempted to solve the CAPTCHA.
> There's a real conflict here. I want Google and Bing and DuckDuckGo to crawl my site and send me new readers. But I don't want everyone else strip-mining it.
Kinda sounds like we're missing a peer to peer network here.
Instead of downloading the same data over and over again we can just download it once and then share it.
Wouldn't that be better for everyone involved?
It would also function as a distributed WayBack Machine, in case anything ever happens to the Internet Archive. (Which I think is desperately needed, bot apocalypse aside.)
Anyway, Google doesn't send traffic to your site any more. Only important sites and obvious scams seem to get indexed.
Common Crawl?
https://commoncrawl.org/
The problem there is trust
You can validate by repeating the work and sign your content.
The domain name system is now just a rent seeking scheme. It was great as a temp solution but over time it has deleted more content than preserved. I might in theory be billed for having a country name but I don't pay for a city, street name, house number or postal code. Online you should be able to move your widget shop to widget street. Can bill people who want to live on real estate street or used car street and/or set some requirements.
Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.
"A $5/month VPS won't cut it anymore."
Are you speaking from experience, or inferring from articles like this?
I serve a static site on the lowest Linode $5/month VPS and it is grotesquely overprovisioned for that use case. It is not the case that every site is getting slammed every second by hundreds of requests per second.
Now, if you have some sort of dynamically-computed website that is generated by a slow scripting language that is poorly optimized and hits the database too many times for a single page, yeah, it doesn't take many RPS to take you out. But that isn't the only option; it's the slowest of the slow options. Realistic, there are plenty of sites that match that description, but I concatenated that many clauses on purpose. Drop any one of them and your personal site will be fine.
It's almost always the DBMS that's the bottleneck when websites drop from traffic. As long as you don't do anything fancier than primary key look-ups you're probably fine though.
I survived handling the search search traffic generated by this thread[1] on PC hardware off a residential broadband connection without any sort of degradation. Only time I've gone offline from traffic was when Elon Musk tweeted a link to one of my blog posts, and that was just a short temporary blip.
[1] https://news.ycombinator.com/item?id=28550764
For a “personal website” or any relatively simple static content site, the DB should not be getting hit much at all. Cache is your friend. For my sites I have a map of all relevant queries and when we do updates we run the relevant queries one time and cache them to both memory and file system.
Should being the operative word. In practice it's both easy and common to have a database do unnecessary work when generating a page, a problem that often only rears its head when under unusual load.
Yeah, IIRC my django site was 3 queries, all correctly indexed, for a main page hit, and 2 for the actual posts. I don't recall the exact perf numbers, but I'm pretty sure it was easily in the 50/rps range for a small dual-CPU host... which doesn't sound like much in "requests per second" but is enough to cover a front-page-HN'ing just fine. And that "rps" was just the database-backed pages, all the static content was served over nginx, so that 50rps is a "real", 50 humans per second rps, not something getting consumed by only two or three humans.
Static files on literally any hardware from the last 15 years on modern server software simply won't get cooked by bots. The network switch will bottleneck you before the server will. Your ephemeral port range will run out before the server will.
Bandwidth is still not free, if you cloud host.
I had a client ask me to look into why their bill spiked 5x (admittedly not by much - from a few dollars to tens). They run a static site, and it turned out someone had inadvertently replaced an image with a massive uncompressed photo. Every unique visitor was caching O(25MB) when they loaded the homepage. Also funny because now everyone has 100Mb+ internet, nobody noticed a few extra seconds on load.
There are lots of solutions to this if you actually need to serve a lot of bytes, but you can get stung even with very simple deployments. Not everyone wants to put Anubis or Cloudflare in front of their site.
Only at the top few cloud providers. Search "Unmetered VPS" and you can find $5 VPSes with no bandwidth bill
Cloudflare offers a free plan that's fantastic. The free plan gives you effectively unlimited DNS/CDN traffic for a normal site, while the main practical cap is 100,000 Worker invocations per day, with 10 ms CPU per Worker request and a 100 MB request body limit. (If that means anything to you).
The next tier up from free is $25/month or $240 per year.
https://www.cloudflare.com/plans/ https://www.cloudflare.com/plans/free/
I use this. Buy a domain, set up a cloud flare pages that builds an Astro page on git push, served statically for free.
Does the free tier support geo-blocks? I'm experimenting with it -- running only a "hello, world" site thus far -- but haven't come across any offer of geo-block capability.
I run my personal website (and a bunch of other websites and services) off a somewhat more expensive but still reasonable VPS (I think 20€ at TransIP - it's so little that I forgot). Load is basically nil most of the time anyway. I think the bot problem is not so bad for personal websites.
Yeah, you just make sure your site is fast enough to handle more than 1 RPS.
But if you want them to actually stop, you can also just serve a little JavaScript page that sets a cookie and refreshes, to anyone who hasn't set the cookie. The DDOS attacker doesn't run JavaScript.
GitHub Pages, GitLab Pages, and Cloudflare Pages are all free. If you have a static site, you don't need to worry about lock-in, since your files are easily portable to a different provider.
I made this tool to try and make it easier for beginners to get something up on the public web for free: https://weejur.com
I host a couple WordPress blogs on Lithium Hosting. It's like $5/mo, fixed cost, and I've never had any issues. If I somehow hit a transfer usage cap, they would just shut my site down until the next month, but I've never had that happen.
All my personal websites are on a single $5/mo OVH VPS and it works perfectly fine.
A $5/month VPS should cut it completely fine unless you're doing something very complicated.
My site's built in Go on a reasonably priced VPS and holds up well. I do geoblock traffic from China and a few other countries as well as IP ranges maintained by Spamhaus.
Static site. If you do go for a WCM of some sort you should make sure that if it has a calendar view of some sort it isn't unbounded but that it only goes as far back as your first post, and only goes as far forward as your latest. That's one of the many ways the poorly-implemented AI systems get stuck.
I have a low cost VPS (actually two in two different pre-AI datacenters) and it runs fine. I use nginx to serve the web pages, and the content is about 99% static content.
The vBulletin and PHPbb style forums have issues with slowdown (I haven’t had a forum since 2015; even back then those forums were overrun with spambots), but static content on a nginx site can be served lightning fast.
If you have a static site, GitHub pages is free
As someone who is moving my static sites over to GitHub pages: While they are free and work really nice, the problem is that GitHub frequently doesn’t deploy updates to the pages.
I have frequently have had to update a GitHub page, push the change, and then GitHub’s actions puke instead of deploying the change. The workaround is that I have a .txt file with a list of GitHub actions which failed, and when GitHub actions fails, I update that .txt file and push the updated site, which GitHub actions will hopefully successfully deploy.
GitHub pages are OK for pages which aren’t updated very frequently, but they are not OK for pages which update frequently.
A $5/month VPS is fine for mostly static content. I’ve got a site that has been pulling millions of viewers and sits at the top of google for its niche for a decade. It is now hammered with bot traffic constantly. But everything is cached so it’s fine.
All of my boxes are cheap VPS. Highly recommend people throw Cloudflare in front of their stuff. I switched all of my load balancers over to there and all of that bot crap went away. That combined with a proper UFW setup keeps the weather clear for me.
Highly do not recommend centralising the internet.
Build a better service or better technology.
If you can't, well then... We're stuck.
Until proven otherwise in your specific case, the better technology is just hosting directly on a VPS without Cloudflare. And if your 5$ VPS is maxed out, while serving any less than 10 requests per second, then you need to optimize your software before considering an upgrade.
Okay but the bots are loose and you have a bandwidth cap on most VPSs.
Because you have no CDN, all the bots are coming directly to you.
Because that's no CDN or distributed WAF, they're hammering your box.
Because your IP is directly exposed, you're also showing up on Shodan.
Your server may have enough CPU and RAM to handle the load, but does your monthly bandwidth allocation with your provider?
Many are finding previous bandwidth limits (starting around 10TB/month) are no longer enough.
Bandwidth is cheap at places that don't call themselves "cloud". You can even get unmetered. 10TB/month is usually enough, even with bots. If you're serving large files, put a simple bot test on the page before the files.
> 10TB/month is usually enough, even with bots.
My single static webpage with no updates in 3 years is doing that, which is (one of the reasons) how I end up where that site (and many others in business and personally) is.
You're chasing a dream for a world that doesn't exist anymore.
I'm on a $4/mo droplet on Digital Ocean and my static site has been just fine. I'm using Caddy and it seems to handle the load like a champ. My site is very lightweight though, so ymmv.
Cloudfare has a very generous plan usually for hobby projects. The article did mention that cloudfare did not work for them, but they have recently introduced few features to block AI crawling as well.
I have a very low traffic blog and I just use the nearlyfreespeech to host it. So far it’s been less than $1
A lot of my sites are on $5 vps and run very well.
Sef-host with tailscale or something similar
Exactly. I've been hosting a site from a spare M1 Macbook Pro with Ngrok. An equivalent bare metal server would be much more expensive.
My sites run on cloudflare workers and static pages for free
Seems about right. I rotated my logs this morning. Most humans go to access.log and most bots go to botpoop.log. This is the line count:
2 is really 1 since a human will grab the CSS file. Most bots do not bother with the style-sheet so that's a 40:1 bots to humans. I could cut that down by blocking data-centers but then I inadvertently block a lot of VPN's which I really don't need to do for a static compressed blog served from ram. The bots just get a TCP Reset but it's still fun to log and study them. The most interesting one I've seen recently is ReadYou which may be a reader but it appears to be much more, possibly acting as a cell phone distributed bot collecting data for a centralized site.[1] - https://nochan.net/logs/access.log
[2] - https://nochan.net/logs/botpoop.log
Is there some existing mechanism already that counts how often an ip only scrapes the page and not the css and then block those origin IPs if it occurs “too often”?
Unfortunately, the best practice is to make css cacheable so you need to keep long histories.
I thought about that but to your point CSS is cachable. In fact I made mine immutable. No I just visually spot patterns and use that to study other facets of the agent, other client headers or lack thereof, supported protocol, accepted encoding and so on.
Maybe it is enough to include some css/js which is served without cache and is loaded after all user visible css/js is loaded. Make it small enough to not cause too much bandwidth for the server and legitimate clients. Then anyone who doesn’t hit that CSS file gets banned.
Possibly. I've noticed however that if people default to reader view they don't always pull the CSS. Seems to vary by client software. I'm weird enough to just stick with visual inspection and add characteristics when botters get too clever. Playing with the botters is almost as fun as Megabonk.
It's a sequence of thin lines, between using Chrome out of the box, to using something like Brave, to using a highly customized Zen Browser, to having ChatGPT look up a particular page for you, to using a small BeautifulSoup/scrapy script to scrape dozens of pages, to scraping the entire web. And at every point on this spectrum it's humans driving a "user agent" tool to make requests and process responses on their behalf.
I also see quite a bit of traffic from China and Singapore. I wonder if it's some scraping for AI training. It doesn't really bother me too much because traffic is still fairly low, but it skews all the analytics for me.
I wonder if the author tried out the recently released feature by cloudfare to block ai bots
https://developers.cloudflare.com/bots/additional-configurat...
Some people don't want to use cloudflare on principle. Like that putting the whole internet behind cloudflare or AWS is a bad thing, in principle.
The experience of browsing the web has really suffered lately. The mandatory 3-4 second "verifying that you're a human" block from Cloudflare seem to show up on more and more websites. Seems like a questionable choice from Cloudflare to teach everyone to associate Cloudflare's logo with high latency...
A 3-4 second cloudflare wait seems good to me at this point. I had to just block any cloudflare requests because some pages that used it would just rev up a cpu core indefinitely and hang the browser.
blocking by geo yeah might work - but what happens when someone is traveling abroad ? they've to use a VPN to access your site ?
my take with all the bots - the web is gonna be a bunch of private walled gardens. with most sites set to no index. you will only discover them via referral from someone real.
What happens when someone is from a shit country?
good question ? I worry about that too since I'm from a poor/shit country that I travel to constantly & live for half the time !!
AFAIK you already have to use VPN if you are e.g. westerner visiting China or Belarus.
Cloudflare's "Verify you are human" captcha is the new cookie banner.
I wonder what the web would be like if we priced bandwidth at the requester point and as you go.
Your comment kind of reminds me of the old slashdot spam-email-solution copypasta - the purpose of it is to highlight how so many sensible sounding ideas just can't work in practice.
Original HN title: "99% of My Website Traffic Is Bots"
Not clear how the author arrived at the precise 99% figure; perhaps "99%" is a figure of speech
"And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers."
"I'm trying to run a business here."
What's the business
(where "business" is defined as "buying and selling")
From https://patronview.com/robots.txt
Perhaps this could be construed as a license, e.g., permitting or prohibiting certain uses of the "content"If, for example, the website operator had enforceable intellectual property rights in the "content", such as copyrights, then perhaps the operator could restrict access to the "content" under the threat of litigation to enforce those rights
Basic questions
1. Is the "content" protected by intellectual property rights, e.g., copyrights
2. Does the website operator have intellectual property rights in the "content", e.g., copyrights
3. Does the website operator have agreements with the rights holders, e.g., granting the operator authorization to restrict access to the "content"
I visited someone's blog and didn't have to solve a cloudflare challenge! That's amazing work
I recently got a surge from Singapore and heard a lot more peeps in my circle of friends saying the same thing...
It'll be the Chinese. The big Chinese cloud providers have data centres in Singapore for some reason. My Chinese mobile phone sends all it's telemetry back to Singapore, not mainland China.
for some reason
It's outside the great firewall. No requirements to hand over SSL keys to China. I worked for companies that ran into these challenges and solutions.
at this point the web is mostly machines politely asking other machines for permission to read each other's content, and humans are the ones triggering the captchas.
Machines don't pay my bills though. If the content was free to share widely, then yea, who cares how it's accessed.
I think it’s wild how AI and data bots are putting certain business models under pressure. We’ve already seen the same thing happen with Tailwind.
This is like technological cannibalism.
I stopped posting to my website. Why should it be so much work to stop this theft?
Would it be helpful to have geofencing and regulation?
I'm working on a web app right now, with the intention of it going to be 100% paywalled. It's 95% complete, but the remaining 5% is just implementing the paywall. In the meantime, the app is live and operational with a fully functional signup. I am deleting about 100 new bot signups per day right now, it is crazy out there.
I'm blocking the Amazon search crawler, anything coming from Googleusercontent, and limiting AI crawlers to search rather than allowing AI assistants. The residential proxy waves are something to behold, but Cloudflare does an okay job catching those in the AI labyrinth. Still, it's all a bit silly, and I can't imagine what large sites deal with when I'm tangoing with this much nonsense on a small tech blog.
Don't. Just serve the page unless it's an unusually expensive page to serve (like search).
Sigh .. same here. I don't write blog posts often (enough) but the ones I do write are from personal experiences and I take a lot of care with them. I look at my logs snd see bots everywhere, but now I just let them get on with it. AI scrapers are different though; they get to read my content which, just for them contains a smsttering of finest digital toxin. A pox on your datasets!
The annoying part is a large percentage of misbehaving bots (not obeying robots.txt for example) are via end user proxies across the world. However most of these aren't doing full-browser loop, so if you are behind cloudflare, you can do non-interactive challenge and that can help quite a bit.
Yes! OP here. I did the non-interactive challenge, and yet all those Chinese bots in my article got through (which surprised me).
Previously, I had done a fair amount of research into how Google's monopoly on web crawling further entrenches their monopoly in the search engine market. You can read more about this here, https://knuckleheads.club, there is a long report from ~2020 or so that explains how it worked at the time. The club is mothballed, I am doing other things with my life, and I'm happy to say that we played a very small role in the DOJ ordering Google to share their crawl data with qualified competitors (a work in progress, but it's progressing).
Chatbots have super charged this dynamic though, to the point that it is showing up in the robots.txt data. The last few weeks I've been having Claude rerun some old analysis of Common Crawl from back then, when I have spare usage and time. What I've found is that you can see pretty clearly the rise in people outright blocking AI chatbot related crawlers likely because of how aggressive they have become.
GPTBot is OpenAI, ClaudeBot is Anthropic, CCBot is Common Crawl, Google-Ext is a way for website owners to indicate they don't want their content to be used for AI, Bytespider is Bytedance, Bing and Google are the last two. Take these numbers with a truck of salt, haven't had time to verify them.It's very clear that website owners do not like getting their content scraped and are indicating to GPTBot et al. that they are not welcome. It's a shame that CCBot is caught in the cross fire, but that's life. Bing and Google are doing just fine though, almost like having significant power in the search engine market gives you an advantage in other markets too. Who knew!
This is a static website running on Cloudflare infra. What on earth costs $90 per month? First optimize your infra before throwing up rules in front of your visitors. I have several websites on Cloudflare too and I don't even check how many million requests I get. Because it does not cost me anything.
> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.
Being self aware does not make it okey. Either you are okey with scraping (like me) or against it. Don't use it yourself and block your site at the same time.
These same people will be crying about how Cloudflare ruins the internet because they get these captchas.
I checked the comments to see if anyone pointed to this: I can imagine so many memes with this line :)
I don't look at the logs of my personal site very often as it is a static site, but just went to check, and yeah, it's almost all ai crawlers. Note sure what is going on here, but I hope this isn't really anthropic:
so they broke into your server?
I despise cloudflare. They've decided that my home IP address is bad, so I have to capchas for most websites. Sometimes on infinite loop and I never get to the website. I even reset my home IP address more than once, but it instantly continues. Especially if I use any VPN, even my work VPN.
I think that asymmetry is why scraping keeps getting worse. The scrapers' costs fell faster than everyone's defenses improved.
There it is. Just like the fckn spammers who ruined SMTP email, scrapers externalize the costs.
Who finances the effort to use residential proxies? That takes a lot of effort, even if it's shoddy
There was a popular pirate streaming stick sold on Amazon that ran a residential proxy by day and did ad fraud clicks by night.
Most people don’t know they’re acting as a residential proxy, lots of devices and apps and free tools install spyware which often includes a proxy script. It’s a really shady market.
> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer. That's usually the easiest way I spot a bot.
> And they were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch.
> But those 4,000 bots were just the warm-up.
I just hate this style of writing like you're on Twitter. Why does the above need to be 3 different paragraphs? A paragraph break indicates a separate thought but the author is still talking about the same data and still making their point. The sentence "But those 4,000 bots were just the warm-up." is effective when still the last line of a paragraph and it signals respect for your readers. I stopped reading after this because it's just a terrible reading experience.
Here's a correct version that doesn't read like the author left for a week to think about what the next sentence would be or having some sort of anxiety-induced mental pause:
> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer, which is usually the easiest way I spot a bot. They were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch. But those 4,000 bots were just the warm-up.
Hey! I'm the OP - thanks for feedback on my writing style. I went ahead and fixed this in the article. It should be updated by the time you read this:
https://patronview.com/news/99-percent-of-my-website-traffic...
And you're totally right: I mostly post on X (nee Twitter) and I probably have ADHD or just a low attention span, so I prefer to read things broken up into paragraphs. But for a smarter audience like this, and that reads long-form blog posts, I should tighten it up.
Thank you for the suggestion. LMK any other edits and I'll be happy to tighten it up.
Glad to hear you’re willing to accept feedback. You maybe don’t have ADHD and sorry for continued advice but you shouldn’t assume you have undiagnosed conditions IMO as it allows you to defer your mistakes from the self. Even if you do have ADHD you can still correct and understand good article structure. I highlighted the last sentence of that paragraph because what is clear from the split sentence style is that you’re writing for impact, this lends well to 140 characters but falls apart in longer form writing but as I stated the sentence is still impactful as you’re saying “but wait…there’s more to this!” Which is very intriguing. I think if you’re comfortable writing that way and it helps you split your ideas and sentences up so each one is impactful, that’s a good thing. But consider that style as a draft and then you can go back and group up your impactful ideas into paragraphs very easily.
For other edits I think my qualm applies to other parts of the article but I found that specific paragraph the most impactful way to illustrate what I was talking about. I leave the rest to you as a challenge. Don’t lose sleep over it, there will be more writing in the future to apply it to.
As for the ADHD stuff and the urge to self diagnose consider something less severe but similar symptoms. Have you considered VAST? Here is a good HN comment briefly detailing it and mentioning a book (titled ADHD 2.0 I believe) that may be more in line. I am not a doctor of course and VAST is rather new.
https://news.ycombinator.com/item?id=49035436
Anyways I will finish reading your article now since you’re so wonderful to take a bit of feedback and be proactive.
pretty crazy that this article is seemingly written by an AI used a detector and it is 95% confident its generated
Most AI detectors are dogshit, though. AI is trained on human writing and tries to imitate it, while these detectors basically flag certain writing styles as "AI-generated."