Continual Learning in Token Space | LettaRESEARCH — DEC 11, 2025Continual Learning in Token Space

BRIEFThe continual learning problem in LLM agents is best viewed through the lens of learning in token space: updates to learned context, not weights, should be the primary mechanism for LLM agents to learn from experience.<br>The biggest gap between AI agents and human intelligence is the ability to learn. Humans continually learn and improve over time, acquire new skills, update their beliefs based on new facts, and modify their behavior to correct for past mistakes. In contrast, most AI agents have an incredible amount of world knowledge, but do not meaningfully get better over time.<br>How do we create AI agents that can continually learn? Traditionally, the concept of “continual learning” for neural networks has been synonymous with weight updates, under the assumption that all learning happens in a connectionist way. The central research questions have focused on catastrophic forgetting (new weight updates causing accidental knowledge loss), and when and how to do weight updates via gradient descent.<br>But there is a disconnect between this traditional framing and the reality of modern LLM-based AI agents. Today’s agents are not just model weights, they are weights plus context. The effective "program" that determines an agent's behavior includes not only the model parameters, but also the system prompts, retrieved documents, tool definitions, and accumulated conversation history. Two instances of the same model, given different contexts, can behave as entirely different agents with different knowledge, capabilities, and personalities.<br>This realization opens up a second axis for learning: rather than updating weights, we can update the tokens that condition the model's behavior. We call this learning in token space . At Letta, we are building towards a future where memories learned in token space become more valuable than the model weights themselves: a future where agents run perpetually, gradually enriching learned context through trillions of tokens of experience data, seamlessly transferring their memories across many generations of models.<br>The limitations of learning in weights<br>Continual learning (updating neural network weights over time as new data arrives) has been studied since the late 1980s. Yet despite decades of research, there's not much to show for it: modern LLMs deployed in production do not continually learn, and their weights are frozen at deployment. The one notable exception is Cursor's tab-completion model which uses online RL to continuously improve based on user feedback, but this form of continual learning operates at the population level, improving the model for everyone rather than enabling individual agents to learn from their own experience. Additionally, it is scoped to a narrow domain: short code completions, not general reasoning and actions.<br>Why has weight-space continual learning proven so difficult? For one, the same techniques that make LLM training so successful don't transfer to the continual learning setting. Pre, mid, and post-training require meticulous data curation and careful human oversight and evaluation - a process that is infeasible to repeat each time an agent needs to learn something new.<br>There is also a deployment problem: whose data do you learn from when you have millions of users? Do you train a separate model for each individual, or mix everyone's private experience into shared weights, risking private data leakage across different users? Efficient methods like LoRA reduce the compute requirements for fine-tuning enough to support personalized, per-use models, but they are still designed for an offline setting with offline evals, rather than continual online updates.<br>Fine-tuning approaches like LoRA also leave harder questions unanswered: where does the learning signal come from, how to weigh recent information against older knowledge, and how to prevent, detect, and evaluate overfitting, distribution shift, and catastrophic forgetting.<br>The illusion of continual in-context learning<br>If the weights of an LLM aren't updated once the model is deployed, how can an LLM agent learn from experience? The primary mechanism for learning online in LLMs is in-context learning: as an agent interacts with the world, its reasoning, actions, and observations are appended to the context window and utilized as new knowledge. This form of “continual learning” works, but has clear limitations:<br>Finite context: Context windows are finite: frontier models cap out at 200k to 1M tokens, and even within advertised context windows, suffer from degraded reasoning, aka “context poisoning” or “context rot”. True continual learning should operate over an infinite time horizon.<br>Append-only structure: Appending raw experience is a poor approximation of learning. When humans learn, we don't just record a log of everything that happened. We create memories, but also refine, consolidate, and...