Evaluating the GPT-5.6 family - Blog - Braintrust

DocsCustomersBlogPricing<br>Contact usSign inSign up

BlogEvaluating the GPT-5.6 family

10 July 2026Izzy Hurley11 min<br>Best practices

The GPT-5.6 family launched yesterday with three models. Sol is the flagship, Terra is the general-purpose middle, and Luna is the fast, cheap one. It also arrives with strong scores on agentic benchmarks. For your work, which one is good enough, and how little can you pay for it?

This eval answers that question directly. I ran the GPT-5.6 family, plus Anthropic's Fable, Opus 4.8, and Sonnet 5, on a set of small, exact, machine-checkable tasks. Then I broke results down by task type and difficulty and turned them into a decision map you can route against.

Eval setup

DimensionValueModels testedGPT-5.6 Sol, Terra, Luna; Anthropic Fable, Opus 4.8, Sonnet 5 (with GPT-5.5 as a baseline)Dataset225 procedurally generated, code-graded tasks (no public-benchmark contamination)Task familiesarithmetic, symbolic_rules, data_transform (75 tasks each)Difficultyeasy / medium / hard, 25 tasks per family × difficultyRuns per task3 independent attempts per modelScoringExact match against the expected answer, checked by code, with no partial creditWhat I reportSolve rate (how often the model gets it exactly right) and cost per call, with confidence intervals1

Every task has one correct, machine-checkable answer. Scoring is deliberately strict. A model that reasons correctly but wraps the answer in the wrong format still fails, because production systems consume model output directly.1 I report both raw solve rate and correct answers per dollar, so you can tell the best model apart from the cheapest model that clears your bar.2

What I'm testing

This eval isolates the small, exact operations agents lean on constantly, like arithmetic, following an ordered set of rules, and reshaping structured data into a required format. These are the steps where one wrong number or a single missing field can break a downstream pipeline.

Task familyWhat it measuresExamplearithmeticMulti-step number crunching with exact outputsReconcile multi-channel inventory and return {total, spread, leader}symbolic_rulesFollowing an ordered set of transformations exactlyApply a rewrite program to a string and return the exact resultdata_transformParsing input, applying business logic, emitting structured outputTurn a row set into a nested JSON decision object

One caveat is worth stating up front. The labels "easy," "medium," and "hard" describe how I built the tasks, not a guarantee that scores fall in that order. Arithmetic is the clearest exception below.

Results

Cost vs. quality

Within the GPT-5.6 family, Sol and Terra are nearly tied, both around 83% overall, while Luna trails at about 68%.1 Anthropic's models look weaker on the headline number (Sonnet 5 63%, Opus 55%, Fable 41%).

That gap is almost entirely refusals rather than wrong answers. On this task<br>style Fable, Opus 4.8, and Sonnet 5 often stop with a refusal instead of<br>attempting the work, and on the attempts they do complete they rank much<br>higher, with Fable the most accurate model in the whole field. The refusal<br>section near the end has the full breakdown.

Where each model does well

Strengths cluster by task type. Sol is strong nearly everywhere, Terra stays close behind, and Luna holds up well on data transforms but falls off on symbolic rules.

The task type matters as much as the difficulty, and the model strengths cluster by task type. Sol is strong nearly everywhere, Terra stays close behind, and Luna holds up well on data transforms but falls off on symbolic rules.

Data transforms are the most forgiving. Even Luna stays strong (87–100%) at every difficulty, so cheaper models are safe here.

Symbolic rules are the hardest to route. Only Sol stays reliable across the board (95–97%); Terra plateaus in the mid-80s and Luna falls apart on the hardest tier. If exact rule-following matters, go straight to Sol.

Arithmetic breaks the expected difficulty order. Every model does best on the medium tier and worse on "easy." The "easy" arithmetic tasks are multi-field reconciliations where one missing field fails the whole answer. They are straightforward to construct but hard to execute.3

Where the model choice matters most

On easy tasks the models bunch together and the choice barely matters. On hard tasks the gap between best and worst blows open. Picking the right model matters most where the work is hardest.

The cross-vendor picture

The family-level results show data transforms are nearly solved for the OpenAI models, symbolic rules pull them apart, and the Anthropic rows are dragged down by refusals rather than by wrong answers.

The decision map

For each task type and difficulty, the map shows the cheapest model that clears a 90% reliability bar (hatched cells mean no model got there):

For data transforms, Luna is the cost-optimal default on easy and medium work, stepping up to Terra only at...