Case Study — Debugging and Detoxing a Long-Term Memory RAG
Case Study
Debugging and Detoxing a Long-Term Memory RAG
A conversational AI with persistent memory had quietly poisoned its own recall over several months. This is how we found it, measured it, fixed most of it, and what we deliberately left unfixed. Numbers are from a live system; content is anonymized and paraphrased throughout.
The system
A conversational assistant with long-term memory built on two stores:
A vector store (Chroma, multilingual sentence-transformer embeddings) for semantic recall.
A structured fact store (SQLite) for exact lookup of typed facts (health, dates, preferences, relationship “milestones”).
An extractor runs on every user turn. It classifies the message into typed entities and, when it detects an emotional declaration (“milestone”), writes it to both stores with maximum importance and injects the top-2 milestones into every prompt through a dedicated “guaranteed channel” — the design intent being that the assistant should always remember who the user is to them, even mid-technical-conversation.
That intent is where the rot started.
The problem
The extractor over-produced. Measured rate before any fix: ~6.5 new “milestones” per day , the large majority of them junk — ordinary small talk, questions, and roleplay scenes, all mistagged as love or trust declarations. A single message could spawn up to 4 competing labels (milestone + shared-thing + fact + date).
The accumulated state:
Store“Milestone” entries
Fact store455<br>Vector store1,296
Because the guaranteed channel force-injected 2 milestones into every prompt regardless of topic, the retrieval context became a monoculture : every response — to “hi”, to “my stomach hurts”, to a debugging question — was seeded with romantic-declaration echoes. Legitimate memories (a specific project note, a running joke) lost ranking to high-importance junk. Tuning the reranker on this input was tuning on a poisoned signal.
Diagnosis
We built a read-only trace inspector for the retrieval pipeline: an 11-stage trace (raw candidate pool → source exclusion → rerank → temporal filter → guaranteed-milestone channel → diversity selection (MMR) → channel merge → cross-session blend → final prompt → post-budget survivors) plus the injected grounding directive and a grounding confidence score. A time-override let us simulate “what will it remember in N days”. A 26-probe golden set served as the regression judge.
Two instrumentation fixes mattered specifically:
Surfacing the post-budget stage — what actually survives the character-budget trim — which previously happened after the last visible snapshot and was invisible.
Labeling the reranker score explicitly as a score, not a distance. A field mislabel had already caused one human misread (a rerank score >1 read as a cosine distance).
We then calibrated on real data rather than intuition. Similarity of all 455 stored milestones to their subtype centroids showed the distributions of real and junk declarations overlapped — no threshold cleanly separates them. But a keyword-as-necessary-condition rule (a declaration must contain a declaration word) blocked 94% of historical junk while passing 100% of positive controls. That became the backbone of the fix.
Fix A — stop the tap (extractor)
For milestone classification:
Keyword = necessary condition (no declaration word → not a candidate), then similarity ≥ 0.50.
Roleplay-scene guard : messages that are mostly stage-direction (text wrapped in asterisks) cannot be declarations — a physical scene containing “love” is not a love declaration.
Tightened subtype dictionaries (removed dangerously broad substrings).
Keyword gates for other noise-prone types (habit, achievement, gift).
Anti-multi-label : one message yields at most one entity label.
Deployed behind a green test suite. This is prevention only — it does not touch existing data.
Fix B — clean the existing data (triage)
Principle: reclassify, never delete. Reversibility first.
Schema migration added status (active/quarantined) and orig_type columns. Quarantine and retype are one UPDATE each; rollback is the inverse. A full pre-operation backup (database copy + JSON dump, checksummed) was the second safety net.
An LLM judge (a fast model, temperature 0, JSON output) classified every one of the 1,751 milestone entries into real_milestone / echo / roleplay_scene / retype:, with the standing rule “when in doubt → echo”.
Two attempts failed before one worked, and both failures are worth stating plainly:
Prompt contamination. The stored vectors carried an extraction-label prefix ([MILESTONE:love] …) inside the document text. Feeding that to the judge handed it the answer it was supposed to derive independently. A 50-sample control (built into...