Is grep really better than a vector DB? · Sara Zan

This is episode 7 of a series of shorter blog posts answering questions I received during the course of my work. They discuss common misconceptions and doubts about various generative AI technologies. You can find the whole series here: Practical Questions.

For the past two years, the default architecture for giving LLMs access to a knowledge base has been RAG with vector databases .

This architecture turned out to be very powerful, but it's far from cheap to setup and maintain: you need the system to chunk all the documents, embed all the chunks, store them in a vector DB, retrieve them, and feed them to the model. Every new document needs to go through this pipeline before it's usable, and changes to a document already processed means going through the vector DB and deleting all the affected chunks.

So it may come as a big surprise for many of us to learn that, as some people claims, an agent equipped with grep can find the data it's looking for just as well.

For example, one of the many comparisons that have been done on this topic found roughly these figures when answering questions about a Django codebase:

Vector search over embedded chunks achieved ~60% accuracy.

Agentic search using tools like grep, find, and cat, where the model iteratively explores the repository, achieved ~68%.

The same test on a TypeScript/Go codebase had the two approaches both reaching around ~70%.

The difference was cost and context : vector search consumed significantly more tokens. While it arguably provided more context to the agent, it's not clear whether the context retrieved this way was more useful, and it's easy to find contradicting results on the Internet.

So, what's truly going on?

If I had to summarize it in one sentence, it would be: retrieval quality depends heavily on the domain and the structure of the information.

Let's check our assumptions

Classic retrieval steps in the RAG architecture assumes the query must be correct on the first try . Because of this, most of the effort and breakthroughs in this field focused on getting decent results for all possible queries.

Vector search shines at this. By getting the chunks of context that semantically come as close as possible to the query (or to an answer to it), embedding-based retrieval was crucial for the one-shot retrieval typical of RAG apps.

However, agents remove that constraint. An agent can try an initial search with a surely subpar query, then inspect the results, refine the query, and search again.

This iterative process dramatically reduces the weaknesses of keyword search and, in fact, leverages all its strengths: exact keyword retrieval is not an ideal task for a vector DB, because semantically similar keywords will also be included.

This does not mean vector search is useless in the age of agents.

Domain Differences

Retrieval strategies behave very differently depending on the domain. Let's see a few examples to understand where one approach or the other shines.

Code Search

Code search is a perfect candidate for agentic keyword search , because identifiers are keywords that need exact matches in the results.

Vector search, while possible, has always been difficult to perform effectively on code. On top of that, there are tons of tools and techniques made for human coders to navigate a codebase with keywords, and agents can take advantage of those. For example, agents can use grep, AST search, symbol indexing, repository graphs, and more.

There's also a problem with context fragmentation . Vector search returns chunks, which in the case of code search are usually a fixed number of lines or symbols. Most of the time this context is useless for the agent, because it rarely includes a full logical unit, and when it does (such as when chunking on function boundaries), it becomes much harder to retrieve, because the chunks are larger.

This means that not only is vector search less precise, but it also wastes a lot of context.

How-to Guides / Knowledge Bases / General Prose

This is the classic use case where vector search shines and keyword search is far less effective.

When your corpus of text is made of conceptual explanations, natural language queries, inconsistent terminology, and so on, semantic similarity is the most likely to bring up relevant results.

Even in this case, however, pure vector search usually gets beaten by hybrid approaches, such as running vector and keyword searches in parallel and then reranking the results.

You can find more about hybrid retrieval in this other post of mine.

Legal / Medical / Scientific Documents

These sit somewhere in between. In these documents, the terminology is specialized, wording matters, and citations and sections are important. Vector search can surface relevant passages, but precision matters more than in the previous scenario. There's more structure than in free-form prose, and you can't lose it during the retrieval...