We gave our agent a memory: building an LLM Wiki over sources that never sit still
Jul 13, 2026 · 9 min read<br>We gave our agent a memory: building an LLM Wiki over sources that never sit still
Our agent kept re-deriving the same answers from scratch, burning minutes and dollars on every question. So we gave it a persistent memory. Here’s how we built it, what we tested, and what changed.
The problem
Our agent answers questions for the team. Ask it something and it goes and finds out: it searches our internal knowledge, our code, Notion, Slack, GDrive, and the web, synthesizes an answer including citations for each claim, and replies back.
Our first approach was standard retrieval-augmented generation (RAG): fetch the relevant raw documents at query time and stuff them into context. The same facts got retrieved and re-derived from the same documents over and over, and answer quality was only ever as good as whatever chunks got pulled into context that particular time.
An alternative is the idea Andrej Karpathy sketched out as an “LLM Wiki”: instead of retrieving raw documents on every query, you compile what you learn once into a persistent, AI-maintained knowledge layer and reuse it. You pay to understand something once, then answer from accumulated knowledge that improves over time instead of re-deriving it from scratch. We’re still early in building one, but here’s what we’ve learned so far.
The part the research hasn’t caught up on yet: our sources move
Here’s the wrinkle. Most of the LLM Wiki work out there assumes a local, personal wiki: your raw sources are files you own, and you know exactly when one is added or edited. Under those conditions maintenance is easy - when a file changes, you just prompt the LLM to recompile the affected pages.
But we wanted a shared wiki compiled from shared sources like Notion, GDrive, or the Web, that are constantly changing underneath us, edited by people who have no idea a wiki depends on them. We get no signal when a source shifts, or when fresh information shows up in a corner we’ve never indexed. Keeping a compiled knowledge layer trustworthy when its underlying sources drift silently turned out to be the hard part, and existing research hasn’t explored it much yet in the context of a LLM wiki.
We haven’t fully solved this yet. Today we lean on two things. A weekly lint job catches contradictions and stale claims after the fact. And more importantly, at query time we re-validate every wiki fact against its linked source before we use it. It means we’re still paying to fetch the source on every answer, a cost the wiki could otherwise reduce.
What our wiki actually is
The wiki is a directory of markdown files living on a dedicated branch of one of our repos: markdown files per entry, each with frontmatter metadata, organized into a nested folder structure. Therefore the wiki is version-controlled, diffable, and reviewable like any other code. There’s no separate piece of infrastructure to babysit.
But it isn’t one flat pile of files, and to see why, start with what happens when a question comes in: we classify it into a category first. Each category has a defined set of sources it’s allowed to draw on.
That classification is what the structure is built around. The first idea was to break one large wiki into smaller sub-wikis, one per category, which makes it easier for the LLM to navigate. Once a question is classified, we route it to the matching sub-wiki, and the agent reads from (and later writes back to) that sub-wiki specifically. This keeps retrieval scoped: a question about one category never has to wade through entries (or sources) that belong to another, which cuts both noise and cost.
Navigation across the structure runs on index.md files. At every level of nesting, the LLM maintains an index.md that summarizes what lives beneath it. To find something, the agent walks the tree top-down: reading the index at each layer to decide which folder or file to open next rather than scanning everything. The indexes are how a growing wiki stays navigable instead of collapsing into an undifferentiated heap of markdown.
Karpathy’s design starts with a dedicated ingest step - you drop sources in and the LLM compiles them into pages up front. Our flow is different: we don’t ingest ahead of time, because our sources won’t let us. They’re shared and constantly changing, so anything we compiled up front might be stale before it gets queried. Instead the wiki fills itself as a byproduct of answering questions. That leaves us with two moves:
Query + Ingest - a person asks something. The agent pulls the relevant wiki entries, confirms they’re still accurate against their sources, and also checks the other shared sources for anything new worth adding, then writes a cited answer, additionally flagging any conflicts it finds along the way. When an answer gets positive feedback, that’s the signal to keep it: we don’t store the response itself, but distil...