5 points | by zionsati a day ago ago
3 comments
https://sr.ht/robots.txt is a robots.txt
Ignoring that AI bot will ignore it, ...
Doesn't "User-agent: AwarioRssBot" needs it's own "Disallow: /" line?
Do LLM crawlers adide by this?
https://sr.ht/robots.txt is a robots.txt
Ignoring that AI bot will ignore it, ...
Doesn't "User-agent: AwarioRssBot" needs it's own "Disallow: /" line?
Do LLM crawlers adide by this?