-->

Deprecated Accessor Broke My Benchmark | Tool-History Footgun - LJ

hack

Skip to main content<br>Companion note: This piece follows We Benchmarked It and HTML as a Native Data Format for LLMs, but this one isn't about markup. It's about a single line of conversation plumbing that quietly decides whether small models can do multi-turn tool calling at all.<br>Here's a transcript from our benchmark. The model has just created a node with a tool call; the system assigned it id m1 and told the model so. One turn later, we ask for the simplest edit imaginable:<br>User: Set the "requireBleed" attribute to true on the node with id "m1" (the widget-slot you created in the previous step). Make the changes with the tools, then reply DONE.<br>Assistant: DONE<br>No tool call. No edit. Just: DONE. A cheerful, confident lie.<br>We had 114 of these (the count from a fresh, fully instrumented re-run of the broken protocol with transcripts captured; the original run had 110, run-to-run variance, same disease). Every one of the 114 was transcript-classified the same way: correct instruction in, zero tool calls out, "DONE." The smaller the model, the worse it got: gemini-3.5-flash failed 96% of these follow-ups, claude-haiku-4.5 failed 71%, while gpt-5.4 and claude-sonnet-4.5 sailed through. We published the finding with a number on it: whole-tree rewrite beat granular tool calling by 33 points on multi-turn edits. We had a tidy story about small models and multi-turn fragility.<br>The story was wrong. Not the data: the data was real, reproducible, temperature-zero real. The interpretation was wrong, and the way we found out is the most useful thing this benchmark ever produced.<br>Results too weird to trust<br>Because the failure mode was so strange (models that execute an instruction flawlessly in turn one, then ignore the identical instruction shape in turn three), we pre-registered a follow-up study to isolate the mechanism. Conversation depth. Mitigation prompts. A fresh-context control. Six models, 2,160 cells.<br>The ablations came back weird. Depth didn't behave: some models got better with more intervening turns, some worse, no coherent curve. A "restate the instruction before acting" mitigation (the kind of prompt hygiene everyone recommends) made gpt-5.4 collapse from 100% to 7.5%. Meanwhile a fresh conversation fixed every model, instantly, universally.<br>Results that incoherent usually mean you're not measuring what you think you're measuring. So I went to read the raw conversation histories we were sending, not the ones I assumed we were sending.<br>The bug<br>Our harness used the AI SDK (v7). After each tool-calling turn, it appended the result to the running conversation like this:<br>messages.push(...result.response.messages); // looks right. isn't.Here's the thing: in a multi-step tool run, result.response.messages contains only the final step's assistant text . The tool calls the model made, and the tool results it received, live one level down, per step. What you actually want is:<br>messages.push(...result.responseMessages); // the idiomatic v7 accessor

// equivalent, built from the per-step data:<br>messages.push(...result.steps.flatMap((s) => s.response.messages));And what makes this a migration trap rather than a plain bug: result.response.messages was the documented v5 pattern for exactly this purpose. v7 kept the property, changed its meaning to final-step-only, and moved the accumulated history to a new top-level accessor, result.responseMessages. It is documented in the migration guide, but silent at the call site. No compile error. No runtime signal. (The property does carry a @deprecated JSDoc note: a strikethrough for humans in editors, invisible to CI, to generated code, and to anyone not hovering.) Code written against v5 still typechecks, still runs, and quietly means something else. Crueler still: the pattern v7 demoted is the very one the 4.0 migration guide told everyone to adopt: 4.0 deprecated responseMessages in favor of response.messages, and v7 moved the accumulated history back to responseMessages. The name has switched meanings twice. We filed this upstream as vercel/ai#16840; the maintainers triaged it as working-as-documented, an enhancement request for guardrails and migration tooling, not a bug. Under the project's definitions, that's a defensible call, and this post isn't an indictment of one SDK. It's about what this class of change (a semantic swap behind a stable, typechecking name) does when it meets agent conversation plumbing.<br>One line. But with the first version, every multi-turn conversation we ran had a hole in it: the model's own tool activity was erased from its history. From the model's point of view, the past looked like this: user asked for an edit; I replied "DONE." No insertNode call. No tool result with the new id. Just a user request and a one-word answer that apparently satisfied everyone.<br>Read that transcript again with the model's eyes and the behavior stops being mysterious. It isn't ignoring...