I worked on large scale RAG systems before and can say people vastly underestimate full text search and vastly overestimate embeddings. FTS is really easy, portable and scalable and gets you very far, the 80/20 rule applies. Embeddings appear to be nice and magic but when you really get into them you notice: semantic similarity isn’t as good as you think and certainly it won’t make everyone happy. You will inevitably end up having to re-embed more or different chunks of your text to accommodate more and more precise embedding search - at which point you’ll go the last mile and do reranking etc etc all the while having to support the operational burden of vector search.
Then you turn around and build a search query with 500 keywords and sure it’s painful but it just works, accommodates all use cases, scales and is overall less annoying to maintain.
I worked on getting an address database into elasticsearch years ago when it was still using modified tf-idf. Customers wanted FTS where a lot of the queries would be something like "100 First Ave, NY" or "200 2nd St, MN".
It was one of the most fun projects I've worked on in my career so far. I got a learn a lot about how US and international addresses worked, so many edge cases, and got to really understand how customers were using the existing search to make sure they weren't adding any duplicates to the database. Token filters and synonyms were neat and figuring out the right indexing strategy was a lot of fun.
It was a lot more work to get it right for most of the use-cases our customers had than just "throw it into ES and be done". That would probably have been fine for the 80/20 case, like you said, but I agree that the bulk of the work is going to be fine-tuning the search solution, whatever technology you're using.
I think people also overestimate the need for full text search when the one doing the querying is an LLM. If your underlying data is structured records, like a customer database, while humans might not have time or skills to figure out that when they want to search by phone number they need to do a join from the contacts table to the users table and normalize the phone number to look up first, making it best to just surface phone numbers as part of the data that is full/text-indexed… an agent is quite happy to handcraft the right SQL to find records that match on a specific field, given the right SKILLS.md and schema information. Turning fuzzy searches into exact DB lookups is a great way LLMs can augment users.
(Obviously this doesn’t apply to searching actual rich document data - for that, go all in on text search, embedding, etc)
How long have "large scale RAG systems" really existed in the first place? I'm always surprised at this, given how new all this really is, relatively speaking.
It's not like a simple embedding search takes that much longer to implement. Especially on short descriptions where you don't have to deal with chunking. And if you let an LLM write the code it's even less of a difference. Combine that with embedding search promising to solve all your search problems, and I understand why people often skip over full text search and go straight to embeddings
I think Bitwarden implemented some vector search in their password search feature ... totally annoying it gives me back all kinds of stuff that I don't care.
I want fuzzy search like 95% of time and then I might consider having additional list of things that can be suggested by vector search.
Bandcamp has had legendarily bad semantic search for as long as they've been around. It's often completely impossible to find an artist or album or song even when you type the exact name.
A good UI could do these and also exact match, give some point system to the results, then order them and perhaps use a bold highlight to reflect what parts of the input query reflected in each result.
On my last go at making my own rag i still got better results by collecting the data and uploading to a project in open(butclosed)ai. My own rag, used by an agent was giving poorer results, and even the agent prefered (derailed)to not use it and look for the info itself rather than using the rag
I would be really grateful if someone could battle-test my frankenstein(lmdb + roaring bitmaps + to-be-removed lance with a bitmap-based virtual fs-like tree on top of your data) outside of its original narrow use-case
And you get bm25 for free with so many modern setups! I do still love to experiment with tuning semantic search for your specific corpus via various kinds of embeddings, but bm25 is hard to beat.
Can you elaborate? We have technicians searching in different languages. Also our knowledge base is often in different languages. I just don't see how full text search can work? Maybe in a problem space like a wiki where people always know what to search for?
Instinctively this feels like a two phase problem - start with some machine translation into a single spoken language and index that, then when people are querying do the same thing. When returning search results show them in the original language.
Yes we've tried. It works. But jargon is hard. RAG with embeddings works all the same. The LLM doesn't mind receiving sources in Italian, french and German, and then outputting the answer in Japanese while providing the verbatim German jargon term in brackets
Embedding search is effectively machine translation into a single common ‘language’ - embedding space - and then searching that; cleaner and less lossy than translating everything into English for searching, but harder to debug when it goes wrong.
RAG is basically good old information retrieval with LLMs doing the querying. This can include vector search but it works without that as well. Treating vector search as magic pixie dust that makes search great without effort is not necessarily going to work that well. Also, it can add a lot of cost and complexity to the equation. And if not tuned properly, you don't necessarily get good results.
The key thing with RAG is to get the right information in the context with as few queries as possible. That requires good recall (ensuring that if it is there it can be found with a reasonable query) and precision (ensuring the best stuff is on top and minimizing false positives).
With search, and by extension RAG, the principle of shit in, shit out applies. Most of what search teams did before AI and RAG is still the best way to optimize the experience with RAG. And if you mess that up, search is not going to be working that well and no amount of AI can compensate for that or only at great cost in tokens and time. So, having an ETL pipeline to pre-process what you index, testing & benchmarking search quality, etc. are all helpful.
The good news is that you don't need that much skills with agentic coding to build something half decent for this. This code almost writes itself. And even a little bit of effort on extracting structure before indexing can make a big difference.
Is anyone else actually finding it harder and harder to read LLM generated text? I find it quite tiring, my brain just does not want to get through it.
the biggest giveway is actually not the writing style, but the content
"using GPT-4o-mini for query rewriting" -> model from 2024, when RAG was trendy, and all the langchain, llama-index, etc, docs mentioned this specific model
Everything that is generate from a LLM is shit, I don't know why people continue using it. I'm waiting for this bubble to explode once for all so we can return doing things in the sane way.
It’s largely because LLMs are reaching for many different types of adjectives or verbs in the same sentence, in a jarring way. While embedding it in a confidently declarative sentence. Everything sounds like some profound insight, dialed to an 11, but written as poetry. Especially those headings. With the short sentences.
The audience for this piece is already very familiar with RAG. I don't want articles discussing e.g. OLED screens telling me what the acronym is - that would be a sign that the article is far below the level that I need.
When I hear stuff like this I always imagine going to a restaurant and asking the waitress for a menu and them replying “lol just google it”.
It’s not that I can’t or don’t know how, it’s rather that the expectation should be that a website should… link you to the information it believes to be relevant background. It’s why it’s called a “web”, linking is a core concept.
> When I hear stuff like this I always imagine going to a restaurant and asking the waitress for a menu and them replying “lol just google it”.
in this case there was a menu in the next empty table and you saw it but in place of getting it you want the waitress to get it for you. Which is a normal behavior but you could save your time by just getting the menu yourself.
No. It isn't. With acronyms, there's often plenty of potential things it can stand for, and if the person doesn't know enough to know which one is the correct acronym, Googling it isn't going to help them.
As OP said, simply providing a link to a Wikipedia article, or a glossary, helps widen the audience beyond "IFYKYK."
The NWS knows this and automatically links to their glossary for both acronyms as well as jargon in their discussions. <-- See what I did there? What does NWS mean in this context? If only I had provided a link that would help you know. I very easily could have. I just didn't.
I thought this would be a useless search that brought up pictures of rags, but indeed, DDG delivers a full page of results about retrieval-augmented generation for the query "rag"
We can confirm, RAG has been a very big thing in the past few years. It's actually bewildering that it be new to some now - but we are also getting the vibe that some are living an ""AI"-nausea" that may be shielding them from some trends.
If like me you run models locally, it's pretty easy to run your own RAG locally also using a Vector Database like Qdrant for persistence, and a middle-layer like Mem0 for realtime retrial and updates. I documented the set-up steps here: https://leadprompt.sh/a/739-Building-an-Infinite-Memory-Loca...
Here’s an even simpler take: just embed everything the first time, then track what was changed. Use a cheap model to summarize and clean up the documents/chats with summary and keywords. Unless you have entire libraries of books to embed it’s going to be a few hundred dollars of API calls.
Then, throw it all in BigQuery. Handles all the vector stuff natively.
Sprinkle an agentic bot UI thing on top to make it appear all-knowing and magical.
I assume other vendors than Google have a similar batteries-included approach you can just plug in.
This assumes your text is small. Try embedding pdf reports - though luck. It surely won’t fit into most embeddings. I can think of many more examples: books, news articles, medical reports, insurance claims etc. they’re all too big to “index it all at once”
How are you gonna handle the relations that span across individual chunks... if a later chunk refers something from 2 chunks before using `it`, rather than proper name, how will you handle that? Because at query time, that later chunk would not match.
What member freakynit said nearby about chunks and relations between chunks, plus the storage and information efficiency problem: make some calculations about storing vectors - for paragraphs and for collections of paragraphs -, then compare the needed space with the original data...
Because you could have clever ideas about vectors related to more paragraphs related in the document structure - but that would multiply the vectors. The index can become much bigger than the corpus.
There have been many blogs like this over the last years.
Yes, embeddings are computationally heavy, but they are not at all complicated and they provide a lot of benefit.
90% of "document" based RAG projects should view semantic search with embeddings as their primary method.
It's very powerful and so easy to implement that you could try it out and discover whether performance would be an issue rather than trying to anticipate it.
Embeddings are reasonably simple, but it’s a journey to get there, and I am very proud of the dog-heavy explainer I wrote on them: https://sgnt.ai/p/embeddings-explainer/
Althought I agree with the first point of the author that FTS is underrated in this new RAG-first framework, the whole article really hides all the problems with RAG-pipeline and kind of hand wave everything.
If you are building a RAG pipeline for your company and are struggling like me, I would recommend this author that has whole series on entreprise documents (start with the one from May 22nd): https://towardsdatascience.com/author/angela.shi/page/4/
Note: I am not the author, just got her article in my newsletter and found it useful.
Does anyone have experience using SMLs for RAG (either as query rewriter or as generator for the final answer)?
I'd like to work with a corpus offline (internal university research data) and I'm hoping I can get everything done without the data leaving the premises.
I guess the biggest bottleneck is going to be for the context window size which won't be able to fit too many result "hits."
Agentic query rewrite on top of good old fashioned Lucene is the end game. This is effectively providing a lot of the same magic you get with the semantic approach. Allowing the agent to query the document store iteratively is where the capabilities become unbounded.
Embeddings and semantic search add non determinism on top of non determinism. This seems fundamentally cursed. Lexical is much easier to control, iterate and debug. The tools are incredibly mature. Your users will probably prefer it as well.
Maybe people are just learning to write in that style LLMs learned to write from statistical people? "is where the capabilities become unbounded" is a weird thing to say and not really true. "is the end game", "add non determinism on top of non determinism", there are a lot of AI-isms in this short comment. But it's possible people are just learning to write this way now, I am curious if that's so too!
As far as uses of time, you are engaging in this dialog too, if you find it not a good way to spend time I recommend ceasing!
The repo includes also plpgsql_bm25rrf.sql : PL/pgSQL function for hybrid search ( plpgsql_bm25 + pgvector ) with Reciprocal Rank Fusion; and Jupyter notebook examples.
I would like to see how each recipe performs against its corresponding evals. Some sort of ranking would be useful.
Everyone keeps posting articles about how to implement RAG, but I also wonder why there isn’t some sort of skill to help people create a simple retrieval plan, starting with the retrieval methods and connecting them with evals. This could show whether they actually improve the result and make retrieval simpler for any agent, instead of making people start from zero.
It seems not many RAG compare themselves across the same benchmarks. https://ggozad.github.io/haiku.rag/ Does an ok job. The part I don’t see being discuss is the whole RL agents writing code to perform RAG queries. It’s one thing haiku-rag does that’s interesting and would like to know what other RAG have that agentic querying with benchmarks
Maybe I'm old but where exactly are the "dragons"?
How is RAG any different from the search systems we've been building before LLMs? Is it the sudden need for everyone to design a search API and engine that's driven this trend?
If so, I'd like to see more design patterns around existing search problems:
- Correcting or backtracking based on feedback.
- Measuring relevance.
- Comparison with task-based pre-written queries. Does every LLM task need a full blown search engine? Why not a tightly scoped domain API for data retrieval?
The whole embedding thing which converts “tokens” to vectors, which you then store in a vector database so that you can later query by vector distance, seems to be LLM specific technology, no? As far as I know the vectors look a lot like the weights in a LLM itself which is why the vector search also works with some level of intelligence.
Vector embeddings predate LLMs. They have been used as far back as the early 2000s. They are a general machine learning technique, rather than LLM specific
Unfortunately LLMs made vector search more popular so it seems like something LLM specific.
What makes it worse, a lot of people in the thread equate vector search with RAG, whereas RAG is the name for anything that model can query so a user doesn't have to copy/paste feed it to the model manually like access to text files is RAG.
Sure, the idea of making a vector embedding for words, sentences, documents etc. is old, but the meat is in how you construct this embedding. I think embeddings have gotten quite a bit better since word2vec.
It's just information retrieval through a new NN based technology that allows to map concepts and ideas as the compression of long text into points in a multidimensional space that manages to compress even more dimensions than the given ones, through non-transparent engines that give different mappings and results, and still (the information retrieval) requires many more clever tricks than the simple idea of vector distance ordering because things do not quite work as they should.
Let's say it's just "computation packaged as something new". "Trivial things".
The article sounds like AI slop with some predictable tells like short punctual sentences, bizarre jargon, and titles like "Recipe 4: On-The-Fly Embedding (The Fresh Data Play)"
Can we not reward junk like this? Most of the sentences are incomprehensible and provide zero actual argumentation, it's just a list of "whats" with no "whys"
I believe embedding-based RAG, everybody is using, will end. As chips advance, you would use a big llm instead of word embedding for retrieval. It's much more accurate and extensive covering every topic.
i want to ask that, if a user want to search sth, but he doesnt know the exact name(keywords), just some description. at this moment, whether the text serach fail?
Ok? I'm not seeing how that is interesting, you're exclusively focusing on coding which requires precise substring locations. Google is basically almost entirely driven by embedding models now.
I worked on large scale RAG systems before and can say people vastly underestimate full text search and vastly overestimate embeddings. FTS is really easy, portable and scalable and gets you very far, the 80/20 rule applies. Embeddings appear to be nice and magic but when you really get into them you notice: semantic similarity isn’t as good as you think and certainly it won’t make everyone happy. You will inevitably end up having to re-embed more or different chunks of your text to accommodate more and more precise embedding search - at which point you’ll go the last mile and do reranking etc etc all the while having to support the operational burden of vector search. Then you turn around and build a search query with 500 keywords and sure it’s painful but it just works, accommodates all use cases, scales and is overall less annoying to maintain.
I worked on getting an address database into elasticsearch years ago when it was still using modified tf-idf. Customers wanted FTS where a lot of the queries would be something like "100 First Ave, NY" or "200 2nd St, MN".
It was one of the most fun projects I've worked on in my career so far. I got a learn a lot about how US and international addresses worked, so many edge cases, and got to really understand how customers were using the existing search to make sure they weren't adding any duplicates to the database. Token filters and synonyms were neat and figuring out the right indexing strategy was a lot of fun.
It was a lot more work to get it right for most of the use-cases our customers had than just "throw it into ES and be done". That would probably have been fine for the 80/20 case, like you said, but I agree that the bulk of the work is going to be fine-tuning the search solution, whatever technology you're using.
What's your opinion on nominatim? I find that it gives up quickly when there's one or two typos in an address. It nails your examples.
I think people also overestimate the need for full text search when the one doing the querying is an LLM. If your underlying data is structured records, like a customer database, while humans might not have time or skills to figure out that when they want to search by phone number they need to do a join from the contacts table to the users table and normalize the phone number to look up first, making it best to just surface phone numbers as part of the data that is full/text-indexed… an agent is quite happy to handcraft the right SQL to find records that match on a specific field, given the right SKILLS.md and schema information. Turning fuzzy searches into exact DB lookups is a great way LLMs can augment users.
(Obviously this doesn’t apply to searching actual rich document data - for that, go all in on text search, embedding, etc)
> people vastly underestimate full text search
It is not psychological, it is fully justified: substring search cannot find synonyms, periphrases and mistaken neighbours.
> It is not psychological, it is fully justified: substring search cannot find synonyms, periphrases and mistaken neighbours.
It is, if people don't even stop to think if they need synonyms, periphrases, or mistaken neighbours.
As the blog post points out, more often than not you don't, particularly if your primary usecase is to search for technical keywords or codenames.
How long have "large scale RAG systems" really existed in the first place? I'm always surprised at this, given how new all this really is, relatively speaking.
I thought text search was always the first thing you try, then fuzzy search, then you go for RAG
It's not like a simple embedding search takes that much longer to implement. Especially on short descriptions where you don't have to deal with chunking. And if you let an LLM write the code it's even less of a difference. Combine that with embedding search promising to solve all your search problems, and I understand why people often skip over full text search and go straight to embeddings
I think Bitwarden implemented some vector search in their password search feature ... totally annoying it gives me back all kinds of stuff that I don't care.
I want fuzzy search like 95% of time and then I might consider having additional list of things that can be suggested by vector search.
Bandcamp has had legendarily bad semantic search for as long as they've been around. It's often completely impossible to find an artist or album or song even when you type the exact name.
A good UI could do these and also exact match, give some point system to the results, then order them and perhaps use a bold highlight to reflect what parts of the input query reflected in each result.
On my last go at making my own rag i still got better results by collecting the data and uploading to a project in open(butclosed)ai. My own rag, used by an agent was giving poorer results, and even the agent prefered (derailed)to not use it and look for the info itself rather than using the rag
I would be really grateful if someone could battle-test my frankenstein(lmdb + roaring bitmaps + to-be-removed lance with a bitmap-based virtual fs-like tree on top of your data) outside of its original narrow use-case
And you get bm25 for free with so many modern setups! I do still love to experiment with tuning semantic search for your specific corpus via various kinds of embeddings, but bm25 is hard to beat.
Yes, and don’t forget, LLMs are very good at tagging, so it’s not even that painful to backfill the corpus.
Can you elaborate? We have technicians searching in different languages. Also our knowledge base is often in different languages. I just don't see how full text search can work? Maybe in a problem space like a wiki where people always know what to search for?
FTS like Elasticsearch supports cross-language (also called multi-language) search.
Instinctively this feels like a two phase problem - start with some machine translation into a single spoken language and index that, then when people are querying do the same thing. When returning search results show them in the original language.
Why not create indexes for multiple languages, as that would also avoid double translation issues (e.g. GER [query] → ENG [index] → GER [document])?
Yes we've tried. It works. But jargon is hard. RAG with embeddings works all the same. The LLM doesn't mind receiving sources in Italian, french and German, and then outputting the answer in Japanese while providing the verbatim German jargon term in brackets
Embedding search is effectively machine translation into a single common ‘language’ - embedding space - and then searching that; cleaner and less lossy than translating everything into English for searching, but harder to debug when it goes wrong.
I don't know what level of quality is required for this site but RAG is trash its just trash. its magic beans.
Very little of this is RAG but rather just FTS with clever reformulation and re-ranking.
RAG is about providing an grounded response, given the actual data in the corpus.
Great article and content, nonetheless!!
RAG is basically good old information retrieval with LLMs doing the querying. This can include vector search but it works without that as well. Treating vector search as magic pixie dust that makes search great without effort is not necessarily going to work that well. Also, it can add a lot of cost and complexity to the equation. And if not tuned properly, you don't necessarily get good results.
The key thing with RAG is to get the right information in the context with as few queries as possible. That requires good recall (ensuring that if it is there it can be found with a reasonable query) and precision (ensuring the best stuff is on top and minimizing false positives).
With search, and by extension RAG, the principle of shit in, shit out applies. Most of what search teams did before AI and RAG is still the best way to optimize the experience with RAG. And if you mess that up, search is not going to be working that well and no amount of AI can compensate for that or only at great cost in tokens and time. So, having an ETL pipeline to pre-process what you index, testing & benchmarking search quality, etc. are all helpful.
The good news is that you don't need that much skills with agentic coding to build something half decent for this. This code almost writes itself. And even a little bit of effort on extracting structure before indexing can make a big difference.
More LLM-generated text about LLMs.
Is anyone else actually finding it harder and harder to read LLM generated text? I find it quite tiring, my brain just does not want to get through it.
the biggest giveway is actually not the writing style, but the content
"using GPT-4o-mini for query rewriting" -> model from 2024, when RAG was trendy, and all the langchain, llama-index, etc, docs mentioned this specific model
Your brain is incredibly adept at pattern recognition; it doesn't focus on LLM-generated text for the same reason it doesn't stare at wallpaper.
We've all learnt that it's not really communication, and so can be dispensed with.
Everything that is generate from a LLM is shit, I don't know why people continue using it. I'm waiting for this bubble to explode once for all so we can return doing things in the sane way.
I'm Becoming AI-Blind: https://news.ycombinator.com/item?id=49386699
It’s largely because LLMs are reaching for many different types of adjectives or verbs in the same sentence, in a jarring way. While embedding it in a confidently declarative sentence. Everything sounds like some profound insight, dialed to an 11, but written as poetry. Especially those headings. With the short sentences.
I have to agree with you. Yet it is tiring, people don't even try anymore.
In the same boat here.
I have a particular antipathy for articles too lazy to spell out acronyms on first use.
So: https://en.wikipedia.org/wiki/Retrieval-augmented_generation
The audience for this piece is already very familiar with RAG. I don't want articles discussing e.g. OLED screens telling me what the acronym is - that would be a sign that the article is far below the level that I need.
A hyperlink to Wikipedia would have solved that issue.
Maybe if a person can't even google RAG they are not the intended audience of that article.
When I hear stuff like this I always imagine going to a restaurant and asking the waitress for a menu and them replying “lol just google it”.
It’s not that I can’t or don’t know how, it’s rather that the expectation should be that a website should… link you to the information it believes to be relevant background. It’s why it’s called a “web”, linking is a core concept.
> When I hear stuff like this I always imagine going to a restaurant and asking the waitress for a menu and them replying “lol just google it”.
in this case there was a menu in the next empty table and you saw it but in place of getting it you want the waitress to get it for you. Which is a normal behavior but you could save your time by just getting the menu yourself.
No. It isn't. With acronyms, there's often plenty of potential things it can stand for, and if the person doesn't know enough to know which one is the correct acronym, Googling it isn't going to help them.
As OP said, simply providing a link to a Wikipedia article, or a glossary, helps widen the audience beyond "IFYKYK."
The NWS knows this and automatically links to their glossary for both acronyms as well as jargon in their discussions. <-- See what I did there? What does NWS mean in this context? If only I had provided a link that would help you know. I very easily could have. I just didn't.
I thought this would be a useless search that brought up pictures of rags, but indeed, DDG delivers a full page of results about retrieval-augmented generation for the query "rag"
We can confirm, RAG has been a very big thing in the past few years. It's actually bewildering that it be new to some now - but we are also getting the vibe that some are living an ""AI"-nausea" that may be shielding them from some trends.
Eh, a healthy web is a web. I enjoy my preferred search engine, but surfing the web is becoming a lost medium.
Hypermedia? In my hypertext markup language?
That is so not Web 5.0. Best I can offer is a support widget that pops up and keeps trying to talk to you until you interract with it.
For those times you need to Red Amber Green your BM25
If like me you run models locally, it's pretty easy to run your own RAG locally also using a Vector Database like Qdrant for persistence, and a middle-layer like Mem0 for realtime retrial and updates. I documented the set-up steps here: https://leadprompt.sh/a/739-Building-an-Infinite-Memory-Loca...
Here’s an even simpler take: just embed everything the first time, then track what was changed. Use a cheap model to summarize and clean up the documents/chats with summary and keywords. Unless you have entire libraries of books to embed it’s going to be a few hundred dollars of API calls.
Then, throw it all in BigQuery. Handles all the vector stuff natively.
Sprinkle an agentic bot UI thing on top to make it appear all-knowing and magical.
I assume other vendors than Google have a similar batteries-included approach you can just plug in.
> embed everything the first time
This assumes your text is small. Try embedding pdf reports - though luck. It surely won’t fit into most embeddings. I can think of many more examples: books, news articles, medical reports, insurance claims etc. they’re all too big to “index it all at once”
you won't get anything out of a whole book embedding anyway, even a structured page is too much
What about splitting bigger content into chunks before embedding?
How are you gonna handle the relations that span across individual chunks... if a later chunk refers something from 2 chunks before using `it`, rather than proper name, how will you handle that? Because at query time, that later chunk would not match.
Humans usually have ways around that in longer documents eg page numbers, paragraphs, links.
If someone gave me a report, in my hands, that said “see ‘it’” I’d also be confused.
What member freakynit said nearby about chunks and relations between chunks, plus the storage and information efficiency problem: make some calculations about storing vectors - for paragraphs and for collections of paragraphs -, then compare the needed space with the original data...
Because you could have clever ideas about vectors related to more paragraphs related in the document structure - but that would multiply the vectors. The index can become much bigger than the corpus.
Yep, lock into some vendor from day 1. Great idea!
Vendor lock in is 2025. Porting became trivial with LLMs advancing like they have.
What I'm saying is pick transportable tech from day 1 so you can easily move if they shut down, hike prices, decide they don't like you, etc.
There have been many blogs like this over the last years.
Yes, embeddings are computationally heavy, but they are not at all complicated and they provide a lot of benefit.
90% of "document" based RAG projects should view semantic search with embeddings as their primary method.
It's very powerful and so easy to implement that you could try it out and discover whether performance would be an issue rather than trying to anticipate it.
Embeddings are reasonably simple, but it’s a journey to get there, and I am very proud of the dog-heavy explainer I wrote on them: https://sgnt.ai/p/embeddings-explainer/
Started reading and will have to finish later but thank you for sharing. Very helpful post.
This is terrific, thank you! There's a typo in the following sentence:
^and^areThis is very good. Thanks.
Althought I agree with the first point of the author that FTS is underrated in this new RAG-first framework, the whole article really hides all the problems with RAG-pipeline and kind of hand wave everything.
If you are building a RAG pipeline for your company and are struggling like me, I would recommend this author that has whole series on entreprise documents (start with the one from May 22nd): https://towardsdatascience.com/author/angela.shi/page/4/
Note: I am not the author, just got her article in my newsletter and found it useful.
Does anyone have experience using SMLs for RAG (either as query rewriter or as generator for the final answer)?
I'd like to work with a corpus offline (internal university research data) and I'm hoping I can get everything done without the data leaving the premises.
I guess the biggest bottleneck is going to be for the context window size which won't be able to fit too many result "hits."
Any info or advice would be appreciated.
Agentic query rewrite on top of good old fashioned Lucene is the end game. This is effectively providing a lot of the same magic you get with the semantic approach. Allowing the agent to query the document store iteratively is where the capabilities become unbounded.
Embeddings and semantic search add non determinism on top of non determinism. This seems fundamentally cursed. Lexical is much easier to control, iterate and debug. The tools are incredibly mature. Your users will probably prefer it as well.
An LLM wrote this comment, no? I'm curious your motivation for having an LLM write such a short comment instead of writing it yourself?
I've not seen such a clipped cadence out of an LLM. I would not automatically suspect the GP. Maybe there's better ways to spend your time?
Maybe people are just learning to write in that style LLMs learned to write from statistical people? "is where the capabilities become unbounded" is a weird thing to say and not really true. "is the end game", "add non determinism on top of non determinism", there are a lot of AI-isms in this short comment. But it's possible people are just learning to write this way now, I am curious if that's so too!
As far as uses of time, you are engaging in this dialog too, if you find it not a good way to spend time I recommend ceasing!
But over-engineering things is fun.
RAG is one of those things where I can hyper optimize to an absolutely needless degree.
If someone has a Postgres db and want very simple RAG:
https://github.com/jankovicsandras/plpgsql_bm25 BM25 search implemented in PL/pgSQL ( Unlicense / Public domain )
The repo includes also plpgsql_bm25rrf.sql : PL/pgSQL function for hybrid search ( plpgsql_bm25 + pgvector ) with Reciprocal Rank Fusion; and Jupyter notebook examples.
I would like to see how each recipe performs against its corresponding evals. Some sort of ranking would be useful.
Everyone keeps posting articles about how to implement RAG, but I also wonder why there isn’t some sort of skill to help people create a simple retrieval plan, starting with the retrieval methods and connecting them with evals. This could show whether they actually improve the result and make retrieval simpler for any agent, instead of making people start from zero.
It seems not many RAG compare themselves across the same benchmarks. https://ggozad.github.io/haiku.rag/ Does an ok job. The part I don’t see being discuss is the whole RL agents writing code to perform RAG queries. It’s one thing haiku-rag does that’s interesting and would like to know what other RAG have that agentic querying with benchmarks
What RAG means for AI is what a library means for human beings.
It's necessary and would be good for you if you want to learn something systematically.
But for most of the normal issues, we can not rely a lot on it.
Agreed, simpler is almost always better. The hard part is resisting the urge to over-engineer it.
> Why this is more flexible than embeddings
Oh boy...
Maybe I'm old but where exactly are the "dragons"?
How is RAG any different from the search systems we've been building before LLMs? Is it the sudden need for everyone to design a search API and engine that's driven this trend?
If so, I'd like to see more design patterns around existing search problems:
- Correcting or backtracking based on feedback.
- Measuring relevance.
- Comparison with task-based pre-written queries. Does every LLM task need a full blown search engine? Why not a tightly scoped domain API for data retrieval?
The whole embedding thing which converts “tokens” to vectors, which you then store in a vector database so that you can later query by vector distance, seems to be LLM specific technology, no? As far as I know the vectors look a lot like the weights in a LLM itself which is why the vector search also works with some level of intelligence.
Vector embeddings predate LLMs. They have been used as far back as the early 2000s. They are a general machine learning technique, rather than LLM specific
Unfortunately LLMs made vector search more popular so it seems like something LLM specific.
What makes it worse, a lot of people in the thread equate vector search with RAG, whereas RAG is the name for anything that model can query so a user doesn't have to copy/paste feed it to the model manually like access to text files is RAG.
Sure and that's a new technique for indexing and querying.
Where's the new design tension? Indexes always had to be monitored for freshness and queries have always needed cleaning or parsing.
right, it is the foundation of machine learning.
not really, vectorising text/books is old school ML by this point.
at least to me that seems the same as https://en.wikipedia.org/wiki/Word2vec for e.g.
Well... Everything new is old "A vector space model for automatic indexing" 1975 - https://dl.acm.org/doi/10.1145/361219.361220
I wonder who was doing doing semantic search in the last century!
"The future is already here—It's just not very evenly distributed..."
Sure, the idea of making a vector embedding for words, sentences, documents etc. is old, but the meat is in how you construct this embedding. I think embeddings have gotten quite a bit better since word2vec.
It's just information retrieval packaged as something new.
It's just information retrieval through a new NN based technology that allows to map concepts and ideas as the compression of long text into points in a multidimensional space that manages to compress even more dimensions than the given ones, through non-transparent engines that give different mappings and results, and still (the information retrieval) requires many more clever tricks than the simple idea of vector distance ordering because things do not quite work as they should.
Let's say it's just "computation packaged as something new". "Trivial things".
And you can't fundraise on some old "information retrieval".
The article sounds like AI slop with some predictable tells like short punctual sentences, bizarre jargon, and titles like "Recipe 4: On-The-Fly Embedding (The Fresh Data Play)"
Can we not reward junk like this? Most of the sentences are incomprehensible and provide zero actual argumentation, it's just a list of "whats" with no "whys"
You are right, now I noticed "Real talk" and "Why this is underrated" and I can't unsee it.
They're absolutely right – and this is is why it's a load bearing observation that cuts to the heart of the issue.
Start with BM25 and only add embeddings when keyword search actually fails you. Saves a lot of pain.
Im curious whether the $10,000 figure includes unstated migration costs, since the raw embedding API cost under the earlier assumptions comes to $10.
I believe embedding-based RAG, everybody is using, will end. As chips advance, you would use a big llm instead of word embedding for retrieval. It's much more accurate and extensive covering every topic.
Still need ~2 years to be replaced.
How would you use a big LLM for retrieval?
Chunk size matters way more than the retrieval model in my experience. Get that wrong and nothing else helps.
Don't leave us hanging! How do you set it?
i want to ask that, if a user want to search sth, but he doesnt know the exact name(keywords), just some description. at this moment, whether the text serach fail?
OT but its interesting that none of the harnesses today use embeddings but just simple grep. I would not have predicted this
Ok? I'm not seeing how that is interesting, you're exclusively focusing on coding which requires precise substring locations. Google is basically almost entirely driven by embedding models now.
And why do you think coding didn’t benefit from embeddings? It was attempted many times and the industry gave up.
I find this interesting because practically no one is doing RAG on thier personal data which is something I wouldn’t have expected.
RAG is so 2024.
Wow! Terrible layout. Shouldn't fully justify on a small screen.