The Hard Parts Of Context Compaction - Benito Lopez<br>Compaction is the practice of making a conversation (often a long one that is close to the context window limit) shorter. It involves summarizing its content and trying to preserve the main information so that an agent can continue the conversation without a loss in performance.<br>Although the basic idea is fairly simple, the compaction process inside a harness has many pitfalls. I encountered them while implementing compaction in ker.<br>ker is still in a fully pre-alpha stage, and this is the first time I have dealt with this problem. So I do not claim to have much authority on the subject. Nor do I claim that ker's current implementation is state of the art. But I believe this article may contain some interesting insights.<br>What compaction is<br>To understand what compaction is, you first need to understand what happens during a conversation with a coding agent. In a stateless harness such as ker, the agent is a loop that accumulates information. Every message from the user or the model, and every tool result, is appended to a list that is sent again in full with every request.<br>The list grows with every turn, while the context window (meaning the maximum number of tokens that a single request can occupy, including the response) never grows. At some point, something has to be cut to avoid exceeding this limit.<br>The process is relatively simple:

Choose a cutoff point in the conversation.

Ask a model to summarize everything that comes before the cutoff.

Put the summary at the beginning of the list, replacing what came before the cutoff.

The more recent part (the part after the cutoff) remains unchanged.

On the next turn, the model will see a summary instead of the user and model exchanges. This should (hopefully) allow the model to continue the conversation in a similar way, without a loss in performance. But it is still a checkpoint that loses information.<br>Own the compaction<br>ker's philosophy is to maintain an audit trail that captures as much of the conversation as possible. This is done by saving everything necessary in the session.jsonl file, so that the entire history can be reconstructed by either a human or an agent.<br>For this reason, the first decision was not to rely on the compaction endpoints offered by providers. As the Earendil team explains well in this post, OpenAI's server-side compaction returns an encrypted compaction item that only OpenAI can decrypt. It can be passed back to OpenAI, but it cannot be used in other providers. So, if I had used this feature directly, I would have ended up with an indecipherable entry in session.jsonl.<br>ker therefore creates the compaction manually by making a request to the LLM.<br>async function compact(history, previousSummary, focus) {<br>// Keep the most recent ~20k tokens verbatim,<br>// everything older is the prefix.<br>const cut = findCutPoint(history)<br>const prefix = history.slice(0, cut)

const message = [<br>`\n${flatten(prefix)}\n`,<br>previousSummary && `\n${previousSummary}\n`,<br>previousSummary ? UPDATE_TEMPLATE : INITIAL_TEMPLATE,<br>focus && `Additional focus: ${focus}`,<br>].filter(Boolean).join("\n\n")

const summary = await model.stream({<br>instructions: SUMMARIZER_PROMPT, // "You are a context summarization assistant…"<br>input: [{ role: "user", content: message }], // exactly one message<br>tools: [], // none<br>})

return [developerMessage(summary), ...history.slice(cut)]<br>}\n${flatten(prefix)}\n`,<br>previousSummary && `\n${previousSummary}\n`,<br>previousSummary ? UPDATE_TEMPLATE : INITIAL_TEMPLATE,<br>focus && `Additional focus: ${focus}`,<br>].filter(Boolean).join("\n\n")

const summary = await model.stream({<br>instructions: SUMMARIZER_PROMPT, // "You are a context summarization assistant…"<br>input: [{ role: "user", content: message }], // exactly one message<br>tools: [], // none<br>})

return [developerMessage(summary), ...history.slice(cut)]<br>}" class="rehype-pretty-copy" onclick="navigator.clipboard.writeText(this.attributes.data.value);this.classList.add('rehype-pretty-copied');window.setTimeout(() => this.classList.remove('rehype-pretty-copied'), 3000);">

The summarizer prompt and the assembly of this request started from<br>pi's compaction prompts; what ker<br>builds around them, how the request is fitted, retried, and committed, is its<br>own.

It is worth pointing out that, at this stage, the model is no longer a coding agent. It has a different system prompt, no tools, and the conversation is flattened into a single message. The model receives a document to summarize. If it received that document as its own history instead, the most natural thing for it to do would be to resume the unfinished work rather than summarize it.<br>Before compacting, throw things away<br>During a work session, most of the context is not conversation, but file contents and command outputs. A read of a thousand-line file takes up as much space as dozens of exchanges between the user and the model. Summarizing all that stuff would be a waste because the model can read those...