Not all model upgrades are upgrades - Microsoft for Developers

Skip to main content

Dev Blogs

AI

All .NET posts

.NET MAUI<br>ASP.NET Core<br>Blazor<br>Entity Framework

C++<br>C#<br>F#<br>TypeScript

NuGet<br>Servicing<br>.NET Blog in Chinese

Microsoft for Developers<br>Agent Framework<br>Develop from the cloud<br>Xcode<br>ISE Developer<br>TypeScript<br>PowerShell<br>Python<br>Java<br>Java Blog in Chinese<br>Go<br>Microsoft Edge Dev<br>Microsoft 365 Developer<br>Microsoft Entra Identity Developer<br>Microsoft Entra PowerShell

Visual Studio<br>Visual Studio Code<br>Aspire

All things Azure<br>Azure SDK<br>Azure VM Runtime Team<br>Microsoft Azure<br>Azure Cosmos DB<br>Azure DocumentDB<br>Azure Data Studio<br>Azure SQL<br>DevOps<br>DirectX<br>Microsoft Foundry<br>Power Platform

OData<br>Unified Data Model (IDEAs)

Windows Command Line<br>#ifdef Windows<br>Inside MSIX<br>MIDI and music<br>React Native<br>The Old New Thing<br>Windows Developer

Waldek Mastykarz

Principal Developer Advocate

A new model drops with lower per-token pricing and better benchmarks. You switch. A week later someone asks why the agent is burning 12x more tokens on the same task while producing worse output.

We ran 150 agent tasks across 15 scenarios on two models, Claude Sonnet 4.6 and Claude Sonnet 5, using GitHub Copilot Chat in VS Code on Windows. The scenarios covered two types of work: architecture and design tasks grounded in Microsoft Learn documentation, and SharePoint Framework project upgrades. Sonnet 5 is the newer model with 33% lower per-token pricing across every token category. The assumption we wanted to test: newer and cheaper means better. What we found was more complicated.

Cheaper tokens, higher bills

Sonnet 5 is cheaper per token across the board. Here’s how the rate cards compare:

Sonnet 4.6<br>Sonnet 5

Input (per 1M tokens)<br>$3.00<br>$2.00

Cached input<br>$0.30<br>$0.20

Output<br>$15.00<br>$10.00

In such a comparison, Sonnet 5 wins every line. But rate cards don’t determine your bill: token consumption does, and Sonnet 5 consumes substantially more tokens.

On architecture tasks (12 scenarios, 60 runs per model), Sonnet 5 used 12x more tokens at the median. One scenario saw a single run consume 47x the typical volume. On code upgrades (3 scenarios, 15 runs per model), the gap hit 10x. A 33% per-token discount doesn’t survive that kind of increase.

What this costs you in dollars depends on the task. On code upgrades, Sonnet 5 cost $2.01 per run versus $0.55 for Sonnet 4.6, making the "cheaper" model 3.7x more expensive. On architecture tasks, the story flipped: Sonnet 5 averaged $0.47 per run versus $0.54, making it 12% cheaper where the token increase was moderate enough for the discount to win out. You won’t know which direction your workload goes until you measure it.

Quality didn’t improve either

The newer model might cost more or might cost less, depending on the task. Does it at least produce better output? On architecture work, based on our evals, no.

Both models completed the task at the same rate, 75% on our Select gate (did the agent attempt the right task at all?). Where they differed was output quality. On the 9 scenarios where both produced usable output, Sonnet 4.6 scored 90% on our Idiomatic dimension (does the output follow established patterns and conventions?) versus 78% for Sonnet 5. The older model outperformed or matched quality in 8 of 9 scenarios.

One scenario, designing an IoT analytics architecture, showed the gap most clearly. Both models completed the task every time, but Sonnet 4.6 passed Idiomatic checks in 4 out of 5 runs. Sonnet 5 managed 1. Same prompt, measurably worse output.

More tokens and worse quality on the majority of scenarios. The "upgrade" went in the wrong direction.

When the upgrade actually matters

Code upgrade tasks reversed the picture. We tested three SharePoint Framework (SPFx) project upgrade scenarios, including the gulp-to-Heft migration and the legacy-to-flat ESLint config migration. Sonnet 4.6 passed the Select gate in 60% of runs. Sonnet 5 passed 100%.

The starkest example was upgrading from SPFx v1.21.1 to v1.22.0. Sonnet 4.6 failed all 5 runs, consistently adopting version 1.22.1 from Microsoft Learn documentation instead of the user-requested 1.22.0. Sonnet 5 followed the instruction precisely every time. When the task requires the agent to follow a specific instruction over what it finds in its context, the newer model was more reliable.

Sonnet 5 also showed a willingness to dig deeper. One run consumed 69 million tokens and met 21 out of 30 evaluation criteria by performing extensive web fetching and discovering undocumented migration steps. Sonnet 4.6 never attempted that depth. The thing is though, that you can’t count on this. 4 out of 5 runs in each scenario didn’t reach that depth either. The breakthrough run is real but not reproducible.

The real ceiling is content, not the model

Both models hit the same quality ceiling on code upgrades. Configuration correctness was 0% across all SPFx scenarios for both models. Neither model completed the structural toolchain...