Interesting solution for sure, just I do not find it scalable for most of the companies. Why instead of installing new software just do not put all LLM required information in llms.txt. If more information is needed put it in llms-full.txt (example: https://modelcontextprotocol.io/llms-full.txt)
The rationale is that with llms.txt you essentially have to dump the whole website info in it, which becomes non-practical for large, deeply nested corpuses.
Authenticated content is another use case. A public text file cannot represent content available only after login.
Interesting solution for sure, just I do not find it scalable for most of the companies. Why instead of installing new software just do not put all LLM required information in llms.txt. If more information is needed put it in llms-full.txt (example: https://modelcontextprotocol.io/llms-full.txt)
I hope this project works, I see the point in it.
Thanks for the feedback!
The rationale is that with llms.txt you essentially have to dump the whole website info in it, which becomes non-practical for large, deeply nested corpuses.
Authenticated content is another use case. A public text file cannot represent content available only after login.
Agentize would cover both of these use cases