Know that feeling of searching for something you know is in your data and not being able to find it? Rechunk is an experiment in how far you can get with embeddings alone — no LLM in the retrieval loop. It dynamically adapts to what the user is looking for, which lets it do things that are hard with fixed-chunk RAG:
- Tailor search to the user
- Retrieve concepts at radically different scales within the data
Know that feeling of searching for something you know is in your data and not being able to find it? Rechunk is an experiment in how far you can get with embeddings alone — no LLM in the retrieval loop. It dynamically adapts to what the user is looking for, which lets it do things that are hard with fixed-chunk RAG:
- Tailor search to the user
- Retrieve concepts at radically different scales within the data
- Detect the _absence_ of a concept
All feedback welcome.