The Curious Case of Agent Memory | Rafi Hasan
← BACK<br>July 24, 2026•12 min read
The Curious Case of Agent Memory<br>Approaches agents use to manage context: compaction, external retrieval, and learned experience.
Table of Contents
I have built static AI agents, where the agent has instructions and tools to fetch information (embeddings, files, etc.) from the vector database and a few other sources. It does not have to update its memory with any new information because the static information doesn't change.
But recently, I have been building this customer support AI agent where it needs to continually update its memory with information about customers, their issues, and resolutions. If Lisa (a customer support agent - real person) helped a customer with a specific issue using this AI agent, and then John (another customer support agent - real person) is trying to help another customer but with similar issues, the AI agent should know what Lisa did and suggest the same course of action to John. In short, the AI agent needs to have memory now.
The Idea
Now what does it mean that an AI agent has memory? Is it sentient? The conversation history is memory. A vector database is memory. A MEMORY.md file is memory. Even a large context window is sometimes sold as memory. But the model does not remember a previous API call. We send it a collection of messages, tool results, and instructions, it produces an output, and that call is over. On the next call, the agent has to assemble everything again.
So where does memory come from?
An agent remembers by rebuilding the right context for its next action.
Deciding what "right" means is the whole problem. If you give it too little, the agent has to repeat work it has already done. If you give it too much, useful information gets buried under old customer records, raw API responses, and conversations. From what I can tell, the state of the art for handling agentic memory falls into three broad approaches: compact the current context, retrieve old information from external storage, and turn past work into reusable experience.
The Context Window
Let's separate the model from the agent. Every large language model has a context window. The agent is the system around it that fills that window. The customer support agent's context might look something like this:
Support Agent Context<br>The Context Window
Focused ContextRaw CRM Export
Window Capacity (128K Tokens)54,000 / 128,000 tokens (42%)
Support Case ContextClick to inspect<br>Support Agent Instructions<br>2,500 tokens(5%)<br>Support Policies<br>4,000 tokens(7%)<br>Current Case Conversation<br>12,000 tokens(22%)<br>Customer & Order Records<br>18,000 tokens(33%)<br>CRM & API Results<br>16,000 tokens(30%)<br>Case Resolution Plan<br>1,500 tokens(3%)
CRM & API Results16,000 tokens<br>CRM searches, payment-provider responses, and knowledge-base results.
💡 Key Takeaway: Everything in this stack is sent to the LLM on every single API call.
All of this is sent to the model as input. If the agent loads a 2,000-line CRM export, those lines now compete with the customer's message, the support policy, and the resolution plan for attention. A one million token context window gives us more room, but it does not decide which 2,000 lines matter. It only postpones that decision.
Prompt caching is useful here, but I don't think it is memory. Caching can make a repeated prefix cheaper to process. It does not choose what the agent should remember.
Approach 1: Context Compaction
The simplest way to keep an agent running is to shrink its history when it gets too large. This is usually called compaction or context editing.
Some content can be removed without much thought. If the agent looks up an order several times, it probably does not need every copy of the same record. The same goes for an early troubleshooting path that the customer has already ruled out. CRM searches and API responses take up a surprising amount of the context, so deleting stale results can buy a lot of room.
Eventually deletion is not enough. The agent has to summarize:
Before:<br>40 messages<br>12 CRM lookups<br>8 payment API calls<br>3 possible resolutions
After:<br>The customer sees two card charges for order #1842.<br>One charge settled. The other is a pending authorization.<br>Do not issue a refund until the pending authorization is rechecked.
The summary becomes a smaller representation of the work so far. It is similar to compressing a log into a checkpoint and continuing from there.
This works well for the current case, but it is lossy. The summary may record that a refund was rejected and leave out the payment status that explained why. It can preserve the final decision but forget a policy constraint that led to it. Once the original context is removed, the agent cannot recover what the summary missed.
There is also a strange feedback loop here. We ask the same model that filled the context to decide which parts of that context future calls will need. Most of the time this is fine. Sometimes the model...