I'm the administrator of 3 online forums, 2 of which are basically archives at this point. All 3 forums were getting absolutely hammered by bots. In fact, all 3 were taken offline by their hosts due to the heavy traffic in 2026. I installed Anubis on 2 of them and bot traffic decreased dramatically to the point where it appears to be only legitimate bots. (Unfortunately, the other forum couldn't use Anubis so easily and has remained offline since then.)
It's clear to me that Anubis won't work forever. You don't need to get as sophisticated as this blog post either as you can bypass the proof-of-work challenge with the right user agent string. I think the reason I'm not seeing bots change their behavior in response to me adding Anubis is simply that my forums are relatively small players. To the people running these scraper bots, a handful of forums adding Anubis is no big deal to them as there are tons of other sites they can scrape.
I think turning the forums that are archives into static HTML is the best path forward for those forums, but that's easier said than done. I've also been thinking about writing a new forum software where all of the content pages are static HTML, and the only dynamic pages are where dynamic features are unavoidable like when posting.
Yeah, I implement PoW on my Edge nodes as well, which connect back to my self-hosted infra over Wireguard tunnels. The PoW works very similarly to Anubis, but practically speaking, they don't really work. Whole swarths of botnets, including botnets that utilise residential proxies (thousands of comprised desktops in poorer countries) just solve the PoW(s) and get through anyway at massive scale.
The only effective way I've found to combat "unwanted traffic" (bots, botnets, crawlers, search engines, etc) is to a) make request/responses as cheap as possible (heavy caching, rate limiting, etc) and b) measure cohorts, fingerprints and behaviour and block patterns that can't possibly look like "normal human traffic" (tricky to get right, but this is to a large extent what Cloudflare is doing I believe, and I've had great success with this technique and building out an OSS Caddy module for this...)
It this point, the best strategy seems to be showing a captcha and only letting through the users that fail on the first try.
Having the users differentiate a scooter from a motorcycle could work too, because every captcha with scooters in it seems to ask the user to select motorcycles.
Yeah I feel static site generators and CDNs are the way. I've been using those for years and it surprises me how people still setup LAMP servers to this day
Same laptop, about 22 seconds in Chrome and about 5 ms with the custom GPU solver. After that, your point about botnets just solving the PoW and getting through sounds pretty believable to me.
I've seen it happen countless times through logs. They just don't care if there's a PoW in the way, they just solve it anyway and move on. When you control a botnet of 10s of thousands of compromised residential machines (residential proxies I believe they're called), you almost have very little recourse except to make it cheap or detect the pattern adn block it (not trivial).
I had recently done some work where I had rehash a bunch of insecure hashes for security reasons. It required brute forcing about 1 trillion SHA-256 hashes to achieve this. Was the first time I had actually ever used Hashcat, very cool piece of software.
I later had a shower thought about Anubis and it's proof-of-work solution to tackle web scrapers. I thought... What if I cracked the hashes out of the browser? Here is my write up of what I discovered.
I'm the administrator of 3 online forums, 2 of which are basically archives at this point. All 3 forums were getting absolutely hammered by bots. In fact, all 3 were taken offline by their hosts due to the heavy traffic in 2026. I installed Anubis on 2 of them and bot traffic decreased dramatically to the point where it appears to be only legitimate bots. (Unfortunately, the other forum couldn't use Anubis so easily and has remained offline since then.)
It's clear to me that Anubis won't work forever. You don't need to get as sophisticated as this blog post either as you can bypass the proof-of-work challenge with the right user agent string. I think the reason I'm not seeing bots change their behavior in response to me adding Anubis is simply that my forums are relatively small players. To the people running these scraper bots, a handful of forums adding Anubis is no big deal to them as there are tons of other sites they can scrape.
I think turning the forums that are archives into static HTML is the best path forward for those forums, but that's easier said than done. I've also been thinking about writing a new forum software where all of the content pages are static HTML, and the only dynamic pages are where dynamic features are unavoidable like when posting.
Yeah, I implement PoW on my Edge nodes as well, which connect back to my self-hosted infra over Wireguard tunnels. The PoW works very similarly to Anubis, but practically speaking, they don't really work. Whole swarths of botnets, including botnets that utilise residential proxies (thousands of comprised desktops in poorer countries) just solve the PoW(s) and get through anyway at massive scale.
The only effective way I've found to combat "unwanted traffic" (bots, botnets, crawlers, search engines, etc) is to a) make request/responses as cheap as possible (heavy caching, rate limiting, etc) and b) measure cohorts, fingerprints and behaviour and block patterns that can't possibly look like "normal human traffic" (tricky to get right, but this is to a large extent what Cloudflare is doing I believe, and I've had great success with this technique and building out an OSS Caddy module for this...)
It this point, the best strategy seems to be showing a captcha and only letting through the users that fail on the first try.
Having the users differentiate a scooter from a motorcycle could work too, because every captcha with scooters in it seems to ask the user to select motorcycles.
Yeah I feel static site generators and CDNs are the way. I've been using those for years and it surprises me how people still setup LAMP servers to this day
Same laptop, about 22 seconds in Chrome and about 5 ms with the custom GPU solver. After that, your point about botnets just solving the PoW and getting through sounds pretty believable to me.
I've seen it happen countless times through logs. They just don't care if there's a PoW in the way, they just solve it anyway and move on. When you control a botnet of 10s of thousands of compromised residential machines (residential proxies I believe they're called), you almost have very little recourse except to make it cheap or detect the pattern adn block it (not trivial).
I had recently done some work where I had rehash a bunch of insecure hashes for security reasons. It required brute forcing about 1 trillion SHA-256 hashes to achieve this. Was the first time I had actually ever used Hashcat, very cool piece of software. I later had a shower thought about Anubis and it's proof-of-work solution to tackle web scrapers. I thought... What if I cracked the hashes out of the browser? Here is my write up of what I discovered.