, description, or og:/twitter: tags here on purpose. Every<br>route declares its own via (src/lib/seo.tsx), which React 19<br>hoists into this head; a static tag would sit earlier in document order<br>and win over the hoisted one. The build-time prerender<br>(src/prerender/) folds each route's tags into its emitted HTML. -->
What actually breaks when you switch LLMs — EvalShift
← all postsmigration·Aug 20, 2026·7 min read<br>What actually breaks when you switch LLMs<br>A model swap is a behavior change with no diff to review. What moves besides the answer text, and why the worst regressions read as correct.
takeaways<br>—A model swap changes behavior — tools called, call order, arguments, output shape, refusals, latency, cost — and ships through a config file with no diff for anyone to review.<br>—The expensive regressions are invisible in the output: an agent that skips issue_refund and still says the refund was processed passes every text-based check.<br>—Migrations usually edit the prompt too, which puts two variables in one comparison — run new model with the old prompt first, then tune, so each result means something.<br>—Hand-written eval cases cover the interactions you already handle; the failures live in recorded traffic you would never have thought to write down.<br>—Significance is not importance: a 0.02 semantic delta can be real and irrelevant, one missing verify_payment call can be statistically invisible and serious. Write the policy before you see the numbers.<br>Changing the model behind an AI feature looks like the smallest change you will make all week. One<br>string moves from gpt-x to gemini-y. The new model is cheaper, or faster, or ahead on the<br>benchmark someone linked in Slack. You try a handful of prompts, the answers read fine, you ship.
The reason this keeps going wrong is that a model swap is not a config change. It is a behavior<br>change, delivered through a config file, with no diff for anyone to review.
## The change surface is bigger than the answer text
Swapping the model can move any of these independently, and most teams only look at the last one:
[role=checkbox]]:translate-y-0.5 border-b-0 px-4 py-2.5 text-caption font-semibold uppercase tracking-[0.1em] text-(--dim)">What moves[role=checkbox]]:translate-y-0.5 border-b-0 px-4 py-2.5 text-caption font-semibold uppercase tracking-[0.1em] text-(--dim)">How it usually surfaces[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">which tools the agent calls[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">a step silently stops happening[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">the order of those calls[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">a check runs after the action it was meant to gate[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">tool arguments[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">right tool, wrong amount, wrong id, wrong units[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">structured output[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">your parser throws, or worse, doesn't[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">refusal behavior[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">the model declines work it used to do[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">verbosity and format[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">downstream regex and UI truncation start missing[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">latency[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">p95 doubles, nobody attributes it to the swap[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">tokens and cost[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">the cheaper model turns out to be the pricier one per task[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">answer quality[role=checkbox]]:translate-y-0.5 px-4 py-2.5 align-top text-body text-(--body)">the only one the playground actually shows you
A migration can improve one row and wreck another. A model that answers just as well but issues one<br>extra tool call per turn is not a cost reduction. You will not learn that by reading answers.
## The worst regressions are invisible in the output
Take an agent that is supposed to do this:
textCOPY ⌘C<br>lookup_order("A-339")<br>issue_refund("A-339", 29.99)
The new model does this instead:
textCOPY ⌘C<br>lookup_order("A-339")
and replies:
Your refund has been processed.
Every text-based check passes. The sentence is fluent, on topic, and exactly what the old model said.<br>The refund did not happen. If you are scoring outputs, this...