Hey, Thanks for the great feedback! You're raising valid point.
Actually, this package started based on a hackathon project where I was burning the Anthropic API credits for our hackathon project which was RAG (internal documentation) + MCP.
There were question which were getting repeated several times. The 50% + comes from this experience. So, based on this, I was thinking of some of the use cases like this:
Multi-User Support/FAQ Systems:
- How do I reset my password?
- Reset password steps?
- Forgot my password help
- Password reset procedure
RAG based:
- How to configure VM?
- How to deploy?
- How to create a network?
You're absolutely right that apps with unique queries won't see these benefits - this won't help in
- Personalized Content
- Real-Time Data
- User-Specific Queries
- Creative Generation
and other scenarios
I think I should clarify this in the docs. Thanks for the great feedback. This is my first opensource package and first conversation in hackernews. Great to interact and learn from all of you
Hey,
functools.cache is definitely simpler and would be sufficient for most basic cases. But I was thinking of multi-tenant and context aware scenario's - that's why went with different strategy.
How is it "50-90%" savings? If a given application doesn't repeat its queries, surely there's nothing to save by caching the responses?
Hey, Thanks for the great feedback! You're raising valid point.
Actually, this package started based on a hackathon project where I was burning the Anthropic API credits for our hackathon project which was RAG (internal documentation) + MCP.
There were question which were getting repeated several times. The 50% + comes from this experience. So, based on this, I was thinking of some of the use cases like this:
Multi-User Support/FAQ Systems: - How do I reset my password? - Reset password steps? - Forgot my password help - Password reset procedure
RAG based: - How to configure VM? - How to deploy? - How to create a network?
Educational/Training Apps Developer Testing scenarios, etc
You're absolutely right that apps with unique queries won't see these benefits - this won't help in - Personalized Content - Real-Time Data - User-Specific Queries - Creative Generation and other scenarios
I think I should clarify this in the docs. Thanks for the great feedback. This is my first opensource package and first conversation in hackernews. Great to interact and learn from all of you
How does this compare to decorating with @functions.cache?
Hey, functools.cache is definitely simpler and would be sufficient for most basic cases. But I was thinking of multi-tenant and context aware scenario's - that's why went with different strategy.
I hate to be that guy, but your AI should have suggested you used one of the off-the-shelf in-memory key-value databases.
The most popular probably being redis.
Fair point! Redis would be better for production. I went with in-memory for zero-config simplicity, but should add Redis as an option. Thanks!