Migrating a production AI agent to GPT-5.6 | Ploy<br>AI Website Builder<br>Visitor Identification<br>Account-Based Marketing<br>Analytics<br>Answer Engine Optimization<br>Hosting

Log inStart Free

As of today, Ploy’s agent runs on GPT-5.6 Sol, the flagship tier of the model family OpenAI released this morning. For months, we couldn’t find a model that challenges Claude Opus given our incredibly high bar for quality. That changed with GPT 5.6 Sol. After running it head-to-head against Claude Opus, we’ve made GPT 5.6 Sol the default model powering every Ploy workspace.

That’s a bigger switch than it sounds. Ploy’s agent builds and edits real marketing websites. It plans a page, reads the codebase, writes components, generates imagery, screenshots its own work, and decides when it’s done. That job description sets a very high bar for a model, and we test every frontier release against it. For the four months Opus held the default slot (first Opus 4.7, then 4.8), nothing we tested beat it. GPT-5.6 is the first model that did.

Not that the first eval run was perfect. It had real failure modes, which we’ll show you. But it did extremely well, and the promise was immediate and specific: builds finishing in less than half the wall-clock time, at 27% lower cost, scoring at or above our incumbent on completed work. Numbers like that buy a model a real migration effort.

Despite using Vercel’s AI SDK, a universal LLM SDK, switching from Claude Opus 4.8 to GPT 5.6 Sol required discovering, one eval failure at a time, that the things we think of as “the model” are provider-specific behaviors our whole stack has quietly specialized around: how it fills in tool arguments, how its prompt cache works, how it replays its own reasoning between turns.

Here’s what it took: fix the eval harness, then the tool schemas, then caching, then reasoning replay.

Step 0: Fix your harness before you trust a single number

Our eval suite runs the real agent against real fixture workspaces. Hundreds of cases, from “build a homepage from scratch” to “is this clone request safe to execute.” Build cases are scored by a visual judge running binary checks against a reference design, ten yes/no questions like “the hero is a full-bleed photographic scene” or “primary CTAs are rounded rectangles, not pills”, plus content checks, tool-trajectory checks, and file assertions. Every failed case gets triaged against its full trace: the actual tool calls and model text, not just the score.

Running that suite across two model families surprised us more than any individual result:

Your harness is tuned to your incumbent model, and you don’t know it. Our tool-call budgets were sized for Opus’s sequential style; GPT-5.6 fans out parallel calls and blew through them on cases it was solving correctly. Our eval executor didn’t support batched file reads, which Opus rarely used and GPT-5.6 uses constantly. Roughly a third of the raw failures in the first cross-model run traced back to harness assumptions, not model behavior, and they were not evenly distributed between the models. If you’re evaluating a challenger model against an incumbent, triage the traces before you trust the pass rate . Otherwise you’re grading the new model on how well it imitates the old one.

Make sure you’re grading models fairly in evals. A dataset that omitted its minScore threshold silently inherited a default of 1.0, so GPT-5.6 “failed” a hero it scored 0.98 on, and Opus “failed” a case while passing every individual check. Two defensible design directions; one invisible threshold.

First impression: immediately promising

With the harness cleaned up, here’s a sample from our redesign suite, where the agent rebuilds a brand’s homepage against a reference design:

Mean per completed buildClaude Opus 4.8 (n=11)GPT-5.6 (n=10)Cost$3.06$2.22 Wall-clock time8m 00s3m 42s Input tokens2.60M1.70M Output tokens33.0K17.1K Visual score0.9360.970<br>This is the shape of the promise: 2.2× faster to a finished page, 27% cheaper, and about half the output tokens. GPT-5.6 writes lean code. On one matched pair, Opus produced a 17,957-character globals.css with 174 CSS variables (full color ramps, mostly unused) where GPT-5.6 wrote 2,508 characters and 45 variables for a comparable (and sometimes better) rendered page.

Claude Opus 4.8<br>See full size<br>GPT-5.6 Sol<br>See full size

Design: sharp, clean, but a little bit uniform

Our overall read on GPT-5.6’s design work: it is very good at clean, modern, tightly-gridded layouts, but it tends to converge towards that look unless you steer it well. With our older harness designed for Opus 4.8, GPT 5.6 Sol tends to ignore existing design systems and instead produces sharp, restrained, and visibly generic output.

The details of how we fixed this are worth a separate blog post of its own. With the expertise of our design and engineering teams, we are able to steer models to achieve world-class brand adherence that you can’t get out of the box.

Step 1: Check your tool...