The Cheap Tier Disagrees With the Expensive Tier

modelsagree .com / labs

experiment 02

Everyone comparing AI models compares brands: ChatGPT versus Claude versus Gemini versus Grok. That's what we do every day at modelsagree.com — re-polling all four on hundreds of “best X” questions and publishing where they agree. But there's a quieter variable hiding under the brand name: the tier picker . Claude ships as Haiku, Sonnet and Opus. Gemini ships as Flash and Pro. Grok ships as Fast and Expert. Most users never touch the default. So we asked: if you hold the brand constant and change only the tier, does “the best vector database” stay the same?

It does not.

The experiment

one prompt · ten questions · every tier we could reach

We took ten live categories from our AI-tooling leaderboards — vector databases, AI coding assistants, LLM observability, RAG frameworks, GPU clouds, text-to-speech APIs, LLM gateways, agent frameworks, eval tools, and frontier API providers — and sent the identical ranking prompt (the same one our site uses, verbatim, top-5 with reasons) to every tier of each family we could access as a normal subscriber:

Claude: Haiku → Sonnet → Opus

ChatGPT: GPT-5.5 → GPT-5.6 (OpenAI blocks the mini tiers on consumer accounts — so for GPT this measures generations, not sizes)

Gemini: 3.5 Flash → 3.1 Pro

Grok: Fast → Expert (grok.com's own reasoning tiers)

Same day, same wording, one answer per tier. Then we compared each family's tiers against each other.

The scoreboard

“did your own tiers crown the same #1?” — out of 10 questions

Claude — Haiku vs Sonnet vs Opustop-5 lists overlap 52% between tiers<br>same #1: 4/10

ChatGPT — 5.5 vs 5.6top-5 lists overlap 64% between tiers<br>same #1: 6/10

Gemini — Flash vs Protop-5 lists overlap 61% between tiers<br>same #1: 5/10

Grok — Fast vs Experttop-5 lists overlap 78% between tiers<br>same #1: 7/10

Read that again: within a single brand, switching the tier changes the #1 recommendation roughly half the time . And the top-5 lists behind those #1s only overlap by about 50–65% (Jaccard) between tiers of the same family.

Across all nine tiers, zero of the ten questions produced a unanimous winner.

Who do they crown in their own backyard?

“best frontier LLM API provider” — the self-preference test

The single most revealing question was the most self-referential one: which frontier-model API should a developer build on? Each tier had to rank its own maker against its rivals. Here's every answer:

“Best frontier LLM API provider” — every tier’s #1

FamilyTierCrowns<br>ClaudeHaikuAnthropic Claude API itself<br>ClaudeSonnetAnthropic itself<br>ClaudeOpusOpenAI a rival<br>ChatGPT5.5OpenAI itself<br>ChatGPT5.6Anthropic a rival<br>GeminiFlashOpenAI API a rival<br>GeminiProGoogle itself<br>GrokFastAnthropic a rival<br>GrokExpertAnthropic a rival

There is no clean story here — and that's the finding. The folk theory says “models shill for their maker.” The tier data says self-preference is tier-dependent and inconsistent in direction :

Claude gets humbler as it gets bigger. Haiku and Sonnet crown Anthropic; Opus — the flagship — hands the crown to OpenAI.

Gemini gets prouder as it gets bigger. Flash crowns OpenAI; Pro crowns Google — the only tier in the whole grid that picked its own maker while its sibling tier picked a rival.

ChatGPT flipped between generations. GPT-5.5 crowns OpenAI; GPT-5.6 crowns Anthropic.

Grok never crowns itself. Both Fast and Expert hand the crown to Anthropic — the only family whose tiers agree on this question, and they agree on a rival.

If you were auditing one of these families for bias by testing one tier, you'd walk away with whichever conclusion that tier happened to hold.

The budget tier buys budget tools

a pattern worth watching

A second pattern surfaced that we didn't go looking for. On “best GPU cloud for training,” every flagship tier — Sonnet, Opus, GPT-5.5, GPT-5.6, Gemini Pro, and both Grok modes — said CoreWeave . The two cheapest tiers in the grid, Claude Haiku and Gemini Flash, both said Lambda — the value-priced alternative. On “best vector database,” Gemini Flash skipped Pinecone (the big tiers' favorite) for pgvector , the free Postgres extension.

It's a hypothesis, not a law — Grok's Expert mode also picked pgvector, so the free-tool instinct isn't exclusive to cheap tiers. But the GPU-cloud split is clean: every budget tier picked the budget cloud, and every flagship picked the flagship cloud. Whether that's training-data skew, a shallower read of the question, or something like taste — the cheap models pattern-matched to the cheap stack.

Why this matters if you sell software

the tier picker is an editorial decision

AI answers are becoming a real acquisition channel — we watch it happen in our own traffic. The industry is starting to monitor “what does ChatGPT say about us?” the way it once monitored Google rank. But almost all of that monitoring hits the flagship API tier.

Free users get the cheap tier. The cheap tier gives...