AI Model Plateau: Build Better AI Infrastructure<br>Engineering<br>August 21, 2026·13 min read<br>The AI Model Plateau: Why Your Infrastructure Matters More Than the Next Release

AI model routing and evaluation now decide your production results. Learn how companies are adapting to make the most of the highly competitive LLM landscape.

Evan MarshallCTO

Frontier models are converging, so routing, caching, and evaluation now decide more of your production outcome than the next model release.

Public benchmarks and token prices cannot predict cost per successful task. Test models against the jobs, constraints, and failure modes in your system.

Keep your stack loosely coupled so you can swap models and providers as their price, speed, caching, and reliability change.

The days of 1:1 model comparisons are over.<br>At one point, it felt like a game of leapfrog, with each release from OpenAI beating the last release from Anthropic, until Anthropic jumped ahead again. Some people stuck to their preference, knowing the latest release would temporarily put them back on top, while others switched back and forth.<br>Now, with open-source options like KimiK3 competing on intelligence, and OpenAI and Anthropic bragging about pricing optimization as often as they do model quality, the field is murkier.<br>Sheer model quality is no longer the differentiator it once was, which means your choice of model isn’t either. The value chain is shifting, and pipelines are increasingly becoming the primary constraint.<br>How do you route work? What do you cache? How do you evaluate the results? How quickly can you adapt your harness when something new or better comes along?<br>The companies that can answer these questions will have better, cheaper results than the companies that can’t, even if they work with the same models and in the same problem space.<br>The best returns will come from investing in your infrastructure and in the continuous optimization necessary to remain loosely coupled as the field keeps changing. In short: Keep your options open, and don’t tie your harness too tightly to one specific model. That’s the TL;DR, but we’ve come to this opinion only after a lot of experimentation.<br>Why we’re writing this : Ito is the only AI code review that actually runs your code (where we use roughly 100 billion tokens a week). The numbers here come from our own experience experimenting with many different models and pipeline configurations.<br>Listen to the labs: Progress is plateauing<br>The vendors describe their own releases in increasingly modest terms over time, and the features they lead with tend to focus on workflow and efficiency rather than on output quality.<br>To be clear: a plateau is not a collapse. Progress is still happening (as our experience switching to Luna demonstrates), but it is becoming more incremental and less paradigm-shifting.<br>Take, for example, Anthropic's Opus 4.8 announcement (published in May of 2026), which says, "Users will find Opus 4.8 to be a modest but tangible improvement on its predecessor."<br>Simon Willison, co-creator of Django, pored through the spec sheet and agreed, writing, "Not much has changed since 4.7. [...] Both the reliable knowledge cutoff and the training data cutoff are January 2026, the same as for 4.7. The context window is still 1,000,000 tokens, and the max output is 128,000 tokens."<br>The big feature accompanying Opus 4.8 was dynamic workflows. Dynamic workflows let Claude plan a job, dispatch hundreds of parallel subagents, and verify the results before reporting back. Cool? Of course. But if the model itself were dramatically better, they wouldn't need workflow tooling to paper over its limitations.<br>The case is similar with OpenAI. In its GPT-5.6 announcement (published in July of 2026), they focus on efficiency rather than capability, writing, "By making every layer more efficient, OpenAI is delivering stronger performance per dollar."<br>Their companion engineering post walks through where the gains came from:<br>Better load balancing across geographies and accelerator types.<br>Kernel rewrites that cut end-to-end serving cost by 20%.<br>Speculative decoding improvements worth more than 15% on token generation.<br>Price cuts are exciting, and they can make a big difference in what users can afford to actually do. But a price cut is not a model improvement, and it results from fixing routing and caching. It’s incremental and iterative.<br>The plateau even shows up within product lines. On Artificial Analysis, the intelligence gap between Luna and Terra is only a few points. Opus to Fable looks similar.<br>7 models are within a few points of each otherThe tiers are converging on quality and separating on cost and speed. Your choice of one model over another is no longer about intelligence gains.<br>Their benchmarks are not your benchmarks<br>Each model provider compares their models against benchmarks to disprove this, to signal that model choice remains the most important decision. But their benchmarks don’t necessarily correlate...