Why Vector Search Alone Isn't Enough: Hybrid Retrieval for RAG - InfoQ
BT
InfoQ Software Architects' Newsletter
A monthly overview of things you need to know as an architect or aspiring architect.
View an example
Enter your e-mail address
Select your country
Select a country
I consent to InfoQ.com handling my data as explained in this Privacy Notice.
We protect your privacy.
Close
Helpful links
About InfoQ
InfoQ Editors
Write for InfoQ
About C4Media
Diversity
Choose your language
En
中文
日本
Fr
Online InfoQ Architect Certification<br>The more senior you become, the fewer people pressure-test your decisions. This 5-week cohort gives you that check.
Register Now.
Online InfoQ Org Architect Certification<br>A practical online cohort for senior architects addressing team topologies, value stream architecture, cognitive load, and architecting for flow.
Register Now.
Online InfoQ AI Engineering Certification<br>A practical online cohort for senior engineers making decisions around retrieval, agents, evals, and AI infrastructure.
Register Now.
QCon San Francisco<br>Learn what's next in AI and software, from teams already doing it.
Register Now.
InfoQ Homepage
Articles
Why Vector Search Alone Isn't Enough: Hybrid Retrieval for RAG
AI, ML & Data Engineering
Why Vector Search Alone Isn't Enough: Hybrid Retrieval for RAG
Jun 02, 2026
17<br>min read
by
Aaditya Chauhan
reviewed by
Srini Penchikala
Write for InfoQ
Feed your curiosity.<br>Help 550k+ global<br>senior developers<br>each month stay ahead.Get in touch
Listen to this article - 0:00
Audio ready to play
Your browser does not support the audio element.
0:00
0:00
Normal1.25x1.5x
Like
Reading list
Key Takeaways
Vector embeddings are approximation engines that are excellent at finding semantically similar content, but systematically weak at distinguishing specific entities like version numbers, error codes, and feature flag names.
Production queries rarely fall cleanly into purely semantic or purely lexical categories; most are hybrid queries requiring both meaning and exact-match, which is where single-method retrieval fails.
BM25 (short for Best Matching 25) is a ranking function that provides the precision that embeddings can't. It uses three mechanisms: inverse document frequency (IDF), which is a weighting of rare distinguishing tokens; term-frequency saturation; and document length normalization.
Reciprocal Rank Fusion (RRF) combines BM25 and vector results without the pain of score normalization. It operates on rank position alone, rewarding documents that both retrievers agree on.
A production retrieval stack is layered. BM25 plus vector search is fused with RRF and is optionally followed by a cross-encoder reranking stage for final relevance gains on a small candidate set.
Your company recently launched an internal omni-search, a single system, developed using Retrieval Augmented Generation (RAG), spanning the company's backlog issues, design documents, launch documents, runbooks, and correction of errors (COEs). Engineers, PMs, and managers query it through an LLM-powered chat UI. Teams also wrap it as an MCP tool, so that their AI coding assistants can pull context directly.
Then an on-call team member in the production support group types: "runbook to enable the payment_v2_enforce feature flag in production" and the chat assistant tells them to disable it instead. Internally, the system ranks documents by embedding similarity.
To the embedding model, the two runbooks look almost identical. They have the same flag name, same service, same vocabulary, and a similar surrounding context. The on-call engineer doesn't see this ranking directly. They see the chat assistant's answer generated from the top-K results the retriever returned (and sometimes the right runbook isn't even in the top-K). The answer is at best diluted and at worst confidently wrong.
Related Sponsors
If you have built a search system using embeddings, this situation might feel familiar. The system gets the big idea right but misses the small, specific details that actually mattered.
The query above demanded two things: a semantic understanding of "feature flag runbook" and an exact match on the operation (enable versus disable). Vector search only handled the first.
This is not a flaw in your embedding model; it is how vector similarity works. Embeddings find things similar to your query, not things that match it exactly. Because retrieval feeds the top-K results into the LLM as context, ranking matters as much as recall.
The right answer being present in the top-K is not enough if the wrong answer is ranked above it. The fix isn't to replace embeddings, but to pair them with classical keyword matching on the actual text so that conceptual relevance and exact term matching both contribute to the final ranking.
Where Vector-Only RAG Pipelines Break
To understand why this situation happens, it helps to zoom out and look at the full pipeline. A RAG...