Why AI Agents Forget by Design | Stan Tyan ⌘CtrlK
You’ve seen this loop. A customer is on their fourth message to a support agent built on a frontier model. They’ve given their account number twice. They’ve explained the issue twice. The agent, helpful and polite, asks one more time: “Could you confirm your account number so I can look into this?” The model isn’t broken. The harness around it didn’t forward the earlier turn. And the API, by design, has no memory of what was said two messages ago.
This is not an edge case. It is the default behavior of every major LLM provider’s API, and it is the single biggest source of production friction in agentic systems shipping today. The reasons are architectural, not accidental, and the cost of that architecture lands somewhere on every invoice and every team’s calendar.
Why AI agents forget by design#
The OpenAI Chat Completions API, Anthropic’s Messages API, and Google’s Gemini API share one property that gets surprisingly little attention in the agent literature: they are stateless. Each call is independent. The server holds no session, no user identity, no record of prior turns. What the model “remembers” is whatever your application chose to put in the messages array for this particular request.
That is the architecture every major provider chose. It is not a limitation they apologize for in the docs. The Anthropic Messages API docs state it plainly: the API is stateless, and the client is responsible for sending the full conversational history on every call. OpenAI’s newer Responses API offers a server-side conversation object as a convenience layer, but the underlying inference call is still stateless. State is something the application maintains around the model, not something the model maintains for itself.
This decision has good engineering reasons behind it. Stateless inference scales horizontally without sticky sessions. It makes load balancing trivial. It avoids a class of memory and security bugs that plague stateful systems. And it cleanly separates the model (a function from input to output) from the application logic (which decides what input to assemble). For the provider, statelessness is the right call.
For you, the person building an agent, it means there is no “memory” feature to enable. There is only the context window you fill on each request, and whatever scaffolding you build around it.
from anthropic import Anthropic
client = Anthropic()<br>model = "claude-sonnet-4-6"
# Turn 1: the user provides their account number<br>response_1 = client.messages.create(<br>model=model,<br>max_tokens=200,<br>messages=[<br>{"role": "user", "content": "My account number is ACME-48201. I can't log in."}<br>],<br>print(response_1.content[0].text)<br># -> "Thanks for sharing that. To help with login issues, could you tell me..."
# Turn 2: a new request, no history passed. The model has no recall.<br>response_2 = client.messages.create(<br>model=model,<br>max_tokens=200,<br>messages=[<br>{"role": "user", "content": "Did the login work yet?"}<br>],<br>print(response_2.content[0].text)<br># -> "I'd be happy to help. Could you share your account number so I can check?"<br>The second call has no idea the first call happened. To the API, these are two unrelated requests from the same key. If your agent wants the model to “remember” the account number, the agent harness has to put it back in the messages array next time. That is the entirety of how memory works at the inference layer in 2026.
The context window is not memory#
The obvious response is: “Fine, we just send all the history every time.” This is what most production agents do, and it works until it doesn’t. There are three reasons it breaks before you finish scaling.
The first is cost. Re-sending 50K tokens of conversation history every turn, across a long session, adds up fast. At Claude Sonnet 4.6 input pricing of $3 per million tokens (as of July 2026), a 100-turn session with 50K average resent context costs $15 in input alone, before counting output or any tool calls. Prompt caching helps for repeated prefixes, but only when the prefix is stable, which it usually isn’t in a real conversation. Caches expire on idle. Caches don’t survive a model swap.
The second is latency. Bigger contexts mean longer time-to-first-token. Users notice the cost in milliseconds. Agents that run hundreds of internal turns notice it in dollars-per-task.
The third is the one most teams underestimate: long context degrades model performance. The Lost-in-the-Middle paper by Liu et al. showed that language models, including the long-context flagships, retrieve information from the beginning and end of a long context far more reliably than from the middle. A fact buried in turn 12 of a 40-turn conversation is, statistically, less likely to influence the model’s answer than the same fact pasted into the system prompt or the latest user message. The model isn’t lying when it says it doesn’t remember. It often genuinely cannot find what’s in front of it.
Bigger...