I let a RAG index rot for 13 months. It grew 5x, 90% failed a check. | ragproof.io

← ragproof.io

I let a RAG index rot for 13 months. It grew 5x, 90% failed a check.

A reproducible teardown of what happens to a vector index after 13 months of<br>real document churn, across pgvector, Qdrant and Chroma.

27 July 2026<br>Rostyslav Myronenko

Why I built this

I build, consult and teach in the GenAI domain, and I'm an active AWS Community Builder in AI Engineering<br>category. One of my friends came to me with some version of "Why retrieval quality of our in-house RAG degrades<br>over time?" - and I didn't have a satisfying answer other than "let me look". The root cause was removal of a<br>bunch of docs belonging to another track, and this situation inspired me to write this post.

I built a version of the worst case on purpose: took a real corpus with<br>real edit history, replayed a year of it the way real ingestion pipelines<br>actually behave (including their bugs), and then measured exactly what's<br>wrong with the result. Engine by engine, against a ground-truth ledger<br>that knows what should be true at every point in time.

The corpus is the GitLab<br>Handbook -<br>a real company wiki, thousands of Markdown pages, years of git history,<br>CC BY-SA 4.0 licensed (per GitLab's own<br>"Our Handbook is open source: here's why",<br>not the repo's boilerplate LICENSE file, which is silent on content).<br>It reads exactly like the internal wiki or Confluence space most RAG<br>buyers actually index. On top of it: 100 clearly-labeled synthetic employee records (Faker, seed=42, not real<br>people) for<br>a GDPR erasure demo, since a real company handbook by itself has no personal data to erase.

Method

T0 → HEAD: picked a commit ~13 months before today<br>(d9dfa4eb7a, 2025-06-02) as the starting index, then walked<br>4,742 real first-parent commits forward to HEAD<br>(6a7e263deb,<br>2026-07-10, the day I started the experiment) - actual edits, deletes, and renames from a real<br>repository's history, not simulated churn.

One embedding pass, three engines: every chunk is embedded exactly<br>once (BGE-M3, fp32, HF revision 5617a9f61b, sentence-transformers<br>pinned to 5.6.0) and the identical vectors are upserted into<br>pgvector (HNSW, cosine), Qdrant (cosine), and Chroma (cosine) - all on<br>documented defaults, no engine given the other's tuning. Docker image<br>digests, client library versions, and the master seed (42) are all<br>pinned and published in the repo.

The ingestion bugs are on purpose. Production RAG post-mortems<br>consistently blame ingestion and change-detection, not the model, for<br>index rot. I modeled four canonical failure modes on labeled subsets of<br>the real churn:

(A) Update-without-delete - re-embed and upsert on edit, but<br>never delete the old chunks. The single most common real bug, and the<br>dominant driver of everything below. Applied to 48.44% of update events.

(B) Async/eventually-consistent deletes - issue the delete call<br>but don't wait for it to actually take effect before considering the<br>job done.

(C) Full re-ingest duplication - one mid-timeline "someone re-ran<br>the whole pipeline" event that re-upserts everything under new ids.

(D) Unhandled renames - index the new path as a new doc, leave<br>the old path's chunks orphaned.

Plus a GDPR erasure demo : Article-17-style "delete this person"<br>requests for 40 of the 100 synthetic employees, issued via each<br>engine's normal delete API, with no manual VACUUM/optimize run<br>afterward - because that's what real teams do: call the delete API<br>and assume it's handled.

Every single mutation - real or injected-bug - is logged to a SQLite<br>ledger with the expected index state per engine. That ledger is<br>published alongside the repo, which is what makes every number below<br>checkable by someone other than me.

Findings

Full scorecard →

Before the percentages, the shape of it: the index went from 33,286 chunks<br>at T0 to 166,947 live chunks at HEAD. Five times the size after 13 months,<br>from a corpus that grew nowhere near that much. The difference is chunks that<br>were re-embedded on edit and never removed.

The headline: across all three engines, 89.73% of what's still live in<br>the index fails at least one check - stale, orphaned or a duplicate . All<br>three engines agree almost exactly with each other, because<br>staleness and orphan status are pure ledger facts, not engine-dependent<br>behavior. That figure is a real set union (chunk_id set operations against<br>the ledger, not a sum of the rows below. Those three categories overlap<br>heavily, so adding their individual percentages would double-count and<br>isn't how this number was computed):

116,798 live chunks are stale. It's 69.96% of the whole<br>live index, or 76.59% of the 152,504 chunks whose source document still exists. Their source<br>document has since been edited and the index still serves the old<br>version. Identical in pgvector, Qdrant and Chroma.

131,241 chunks (78.61%) are orphaned - their source doc is<br>deleted,<br>renamed or superseded by a newer version that was never cleaned up.<br>Breakdown:...