Your RAG system isn't hallucinating. It never found the answer. | Artifipedia
Skip to content
MapBlog
Fields
Home/Blog/Your RAG system isn't hallucinating. It never found the answer.
July 15, 2026 · Artifipedia · RAG, Embeddings, Vector Database
Your RAG system isn't hallucinating. It never found the answer.
When a RAG system gives a bad answer, almost everyone blames the model. Usually the right passage was never retrieved — and that changes everything about how you fix it.
ShareXLinkedInCopy link
Here's a scene that plays out in a lot of teams.<br>You've built a RAG system over your company's documentation. It demos beautifully. Then real users arrive, and the answers get strange — vague, subtly wrong, occasionally confident about something that isn't in any document you own. Someone says the word "hallucination." The fix list gets written: try a better model, tighten the prompt, lower the temperature, add "only answer from the provided context" in bold.<br>None of it works. It doesn't work because the diagnosis was wrong.<br>The model didn't hallucinate. It answered faithfully — from material that didn't contain the answer, because your retrieval never found it.<br>Two halves, one blamed<br>RAG is two systems wearing one name. The retrieval half searches your documents and picks passages. The generation half reads those passages and writes an answer.<br>Only one of those halves is glamorous. All the demos, all the model announcements, all the conversation is about generation. Retrieval is a search engine with a new coat of paint, and nobody's excited about it.<br>But look at the failure. If retrieval hands the model three passages that don't contain the answer, what should a well-behaved model do? Say it doesn't know — which most will, some of the time. Or produce something plausible from adjacent material, which is what actually happens when the passages are nearly relevant. That output looks exactly like a hallucination. It has the same shape: confident, fluent, wrong. And it's not a generation failure at all.<br>The hallucination diagnosis is seductive because it's a known phenomenon with a known name. That's precisely why it gets over-applied. You can't fix retrieval by adjusting the model, and every hour spent on the model is an hour the actual bug survives.<br>Test the halves separately<br>The single most useful thing you can do to a RAG system costs an afternoon and no money.<br>Take thirty real questions — real ones, from actual users or realistic ones from someone who knows the domain. For each, note which document actually contains the answer. You now have a retrieval test set.<br>Run just the retrieval. Not the model. Look at the passages that come back and ask one question: is the answer in here?<br>That's it. That's the test. And it's the test almost nobody runs, because retrieval feels like plumbing and plumbing feels like it works.<br>The number that comes back tends to be sobering. If retrieval finds the answer 60% of the time, then your system has a hard ceiling at 60%, and no model on the market moves it. Swapping to a better model when retrieval is at 60% buys you a marginally better-written wrong answer.<br>If retrieval is at 95% and answers are still bad — now you have a generation problem, and now the fix list makes sense.<br>Why retrieval fails<br>Once you're looking at retrieval, the failures are specific and mostly fixable.<br>Chunking cut the answer in half. Documents get split into pieces before they're indexed, usually at some fixed size. If the answer spans a boundary, no chunk contains it, and no search can find what isn't there. A table split from its header. A policy split from its exception. This is the most common and most invisible failure, because everything looks fine in the code.<br>Semantic similarity isn't relevance. Embeddings find passages that are about the same topic. That's not the same as containing the answer. A question about your refund window will happily retrieve five passages that discuss refunds warmly and specify nothing. They're similar. They're useless. The system did what it was designed to do.<br>The vocabulary doesn't match. Users ask in their words; documents are written in yours. "Can I get my money back" and "reimbursement eligibility criteria" are the same question and are not close in embedding space. This is the case where old-fashioned keyword search often beats semantic search outright, and where combining both beats either.<br>One embedding model for everything. General-purpose embeddings were trained on general text. If your domain has its own language — clinical, legal, industrial, internal jargon — similarity in that space may not track similarity in yours.<br>Too many passages. This one's counter-intuitive, so it survives longest. Retrieving twenty documents instead of three feels safer — surely more context helps? It usually doesn't. Models attend unevenly across long inputs, and a correct passage buried in the middle of a large context can be effectively invisible. You've retrieved the...