Remembering Without Room: A Memory Design for Local Small Language Models

Remembering<br>Without Room: A Memory Design for Local Small Language Models

A design viewpoint paper

James Sparenberg Brain-Research Project August 2026

Download this paper<br>PDF<br>Word (.docx)

Abstract

Small language models running on local hardware have a memory problem<br>that big hosted models can paper over. When the context window is a few<br>thousand usable tokens, an agent forgets what it decided ten turns ago,<br>and the usual fix — stuff the whole transcript back into the prompt<br>every turn — makes accuracy worse, not just slower. This paper argues<br>that the answer is not a bigger window. It’s a memory layer that keeps a<br>short sketch in the prompt, keeps the evidence on disk, and keeps a<br>working path between the two. This paper proposes a design built from<br>four modules and four distinct ways of getting at stored information,<br>and it argues that one constraint usually treated as a weakness in local<br>deployment (slow disk access) is actually the thing that makes a better<br>design affordable. This paper also lays out how one would know if any of<br>this works, and what would tell developers to stop.

1. The problem is<br>not forgetting. It’s hoarding.

Watch an agent forty turns into a coding job. It has read nine files,<br>run a dozen commands, and it’s about to ask you a question you already<br>answered on turn four [16].

Nothing malfunctioned. That’s the design working as built. An agent<br>is a stateless model wrapped in a loop that hands it the transcript<br>again every single turn. Nothing persists on its own. So the loop<br>compensates by pasting everything back in: files, tool output, error<br>messages, all of it. Cost climbs with the square of turn count, because<br>turn forty re-sends turns one through thirty-nine.

The obvious fix is a bigger window. Buy more room and stop<br>worrying.

That fix doesn’t work, and the reason is the interesting part.

Chroma tested eighteen frontier models on tasks a child could do [2].<br>Copy this text. Find this sentence. As the input got longer, the models<br>got worse at it. Not at the edge of the window either. A 200,000-token<br>window can start degrading somewhere around 50,000. The number printed<br>on the box isn’t the number you get in practice.

If the only cost of a long prompt were money, a bigger window would<br>solve everything. Accuracy falling before the window fills is a<br>different kind of problem. And for a 3B model with maybe four thousand<br>tokens of reliable attention, the degradation zone starts almost right<br>away.

So the agent that remembers by hoarding gets slower, more expensive,<br>and dumber, all at once.

2. What the field already<br>knows

Two findings from the 2026 survey literature shaped this proposal<br>more than anything else.

First, from a survey covering work from 2022 through early 2026<br>[1]:

The gap between “has memory” and “does not have memory” is often<br>larger than the gap between different LLM backbones. Investing in memory<br>architecture can yield returns that rival — or exceed — model<br>scaling.

The ablations behind that claim are stark. Strip reflection out of<br>Generative Agents and coherent multi-day planning collapses into<br>repetitive noise inside 48 simulated hours [5]. Take the skill library<br>away from Voyager and it reaches tech-tree milestones 15.3 times slower<br>[18]. On MemoryArena, swapping an active memory agent for a<br>long-context-only baseline drops task completion from over 80% to about<br>45% [15].

For anyone building on small local models, that’s the whole argument.<br>A small model with excellent memory isn’t a compromise. It’s a<br>strategy.

Second finding, same survey: long context is not memory. Models that<br>score near-perfectly on passive recall benchmarks like LoCoMo [17] fall<br>to 40–60% on MemoryArena, where memory has to actually drive decisions<br>across linked sessions. Passive recall aces make poor memory agents.<br>That’s an uncomfortable result for anyone planning to test a memory<br>system by asking it to retrieve facts, and take it as a warning about<br>then evaluation plan.

The survey also names the hard open question directly: when does an<br>episodic record graduate into a semantic fact, and when does a semantic<br>fact get pulled back into working memory for a specific job? Most<br>systems, it says, implement two layers well and handle the transitions<br>between them with crude heuristics. That’s not a solved problem one can<br>go look up. It’s the frontier.

3. The design

Four modules. Each one useful on its own, testable on its own, and<br>able to be thrown away on its own if it turns out to be wrong.

3.1 Working memory

Every tool call appends to a plain append-only ledger on disk. The<br>full output never enters the prompt. What goes into the prompt instead<br>is a compact sketch built by the harness, not by the model: what was<br>tried, what worked, what blew up, what it learned. Every node in that<br>sketch carries an ID that points back at the ledger record.

Two rules make this work.

The sketch is capped in tokens, hard....