2 points | by jdauriemma 8 hours ago ago
3 comments
Not at all, only web crawlers from hetzner network.
https://news.ycombinator.com/item?id=46058273
I have seen something similar in a site I work on
There (1) is more context on this topic, where author of LLMs.txt idea said 'llms.txt files have nothing to do with crawlers or big LLM companies. They are for individual client agents to use.'
1. https://news.ycombinator.com/item?id=47058870
Not at all, only web crawlers from hetzner network.
https://news.ycombinator.com/item?id=46058273
I have seen something similar in a site I work on
There (1) is more context on this topic, where author of LLMs.txt idea said 'llms.txt files have nothing to do with crawlers or big LLM companies. They are for individual client agents to use.'
1. https://news.ycombinator.com/item?id=47058870