Thomas Schranz 🍄 on X: "🥪 sandwich truncation

when an agent uses a tool

the output of the tool typically becomes part of the chat history

why? because using tools is the main way for the agent to explore its environment and to interact with it

the output of the tool gives it useful feedback

the" / X<br>Post

Log inSign up

Post

Thomas Schranz 🍄 on X: "🥪 sandwich truncation

when an agent uses a tool

the output of the tool typically becomes part of the chat history

why? because using tools is the main way for the agent to explore its environment and to interact with it

the output of the tool gives it useful feedback

the"

Thomas Schranz 🍄

@__tosh

🥪 sandwich truncation

when an agent uses a tool

the output of the tool typically becomes part of the chat history

why? because using tools is the main way for the agent to explore its environment and to interact with it

the output of the tool gives it useful feedback

the chat history (including the tool outputs) is what gets carried forward

and sent to the language model again and again

so the language model can give the agent its next action steps in return

e.g. more tool calls or a response to the user

but the working memory of a language model is limited ("context window")

it makes sense to be aware of what ends up in the context window in the first place

(system prompt, agents md files, skills, tool descriptions, the messages you are sending, the files you tell the agent to read, …)

the output of a tool usually is valuable context for the agent

with well designed command line tools the output often is fairly terse and token efficient

(e.g. a tool call can be a pipeline of various command line tools doing a lot of work and in the end returning simply with 'done')

but some tool calls can produce a lot of output

(install scripts, test suites, linters, http requests, failed tool calls that produce stack traces, …)

some tool calls can produce so much output that it fills up the context window completely

one pragmatic way to deal with these cases is to change the implementation of the tool itself

that works fairly well with some tools where you can control what output gets produced and how it should get represented to the agent

that said: with more open ended tools like sh / bash / exec this is not so easy to do well

that's why agent harnesses often automatically truncate the tool output in some way

pi and opencode truncate the head of the output and keep the tail

this is very pragmatic

think of the usual command line output

(test suite or install script that produces 1000s of lines of output and in the end show if everything is 'done' or if there were errors)

if you could only see the first 10 lines

or the last 10 lines

which would you pick?

if you can have only one or the other

i guess the last 10 lines are often more useful

but for some command line output perhaps the beginning of the output is more important than the end

e.g. think of reading a csv file where the first line is a header that tells you about each column in the file

or a deploy script that first lists things like the commit id, region, server name and so on

but then again that context perhaps is self-evident based on the rest of the chat history

and it's probably more important to know whether the deploy worked in the end or not

for coding agents keeping the tail and truncating the head is a sensible default

both pi and opencode retain about 50k chars of the tail

in addition to that they also log the full output

and as part of the truncated output they write down where the agent can find the log

codex and hermes are also truncating tool output

but instead of truncating the head and retaining the tail

they are truncating the middle

and keep both the head and the tail

a bit like keeping the top and bottom of a sandwich and removing a bit in the middle

like with pi and opencode if the output is large it gets truncated

but if there is also important stuff in the beginning and not only in the end the chat history will have it (and the agent doesn't have to fetch it from a log file)

kinda neat

a bit like a book or a movie

you remove a bunch of stuff in the middle

and you still have the beginning and the end

now what's better?

none of this is free of course

if you keep 50k chars at the end and nothing of the beginning you have no beginning

if you keep 25k chars of the beginning and 25k of the end

you have both the beginning and the end

but you have less of the end

only half of the end!

didn't we say usually the end is more important?

codex and hermes do sandwich truncation

but they split head and tail in a different way

codex splits 50% head, 50% tail

hermes splits 40% head, 60% tail

sounds pragmatic

is there a right way or a wrong way?

is retaining the tail better?

is keeping the head and the tail better?

what if we keep ~60k chars or only ~30k chars?

what if there are smarter ways of pre-processing the output before it gets...