Basecamp Bench | Stephen M. Walker II<br>SMW.AIAbout

BlogBasecamp Bench

GPT-5.6 Sol, Fable 5, and Grok 4.5 agency, cost, and quality

GPT-5.6 Sol arrived with OpenAI claiming stronger frontend engineering and greater token efficiency than GPT-5.5, while Anthropic released Fable 5 at a dramatically higher price than the other models. SpaceXAI launched Grok 4.5 in the same period as a comparatively inexpensive, Opus-class model that, in theory, should be extremely capable.

I wanted a way to test those claims through real-world use. I chose one sufficiently complex software problem and dissected it across 20 dimensions to see where the models excel, where they fail, and where their performance is nearly indistinguishable.

Benchmark setup and results

Fable 5 frontend prototype showing global search in its Basecamp implementation

I built Basecamp Bench to measure this. You can read the full report here. Agents start in the same greenfield workspace with AGENTS.md, DESIGN.md, INIT.md, and Basecamp 5 reference material. They then build a frontend from mocks (screenshots) and design tokens, and a backend implementing the canonical API contract, revealing their engineering choices and tradeoffs.

Fable 5 won both tracks. Grok 4.5 delivered the strongest speed and cost tradeoff, completing both builds in 37 minutes for $9.30, but with significant polish gaps.

Grok 4.5 frontend prototype showing the sample Basecamp project

ModelFrontendBackendTotal timeTotal costFable 57.5788.3922:06:40$85.87Sonnet 56.9827.2431:27:09$36.23Grok 4.56.3847.27836:48$9.30GPT-5.6 Sol5.7657.31059:48$15.13GPT-5.55.6707.08444:14$10.94

Agent separation

Backend scores cluster tightly because most models register the complete route surface, with invariant enforcement, contract-correct responses, and honest failure handling separating them. Frontend scores vary more as models trade breadth for depth and struggle with polish: the spacing, icons, micro-interactions, and transitions that make an interface feel finished.

Backend benchmark scores across evaluation dimensions

Fable 5 follows the specification most closely, connecting every surface and endpoint to the same product model. Sonnet prioritizes working, persistent workflows, GPT-5.5 prioritizes polish (though not particularly well) while leaving core controls inert, and GPT-5.6 Sol pairs a disciplined backend with a shallow frontend.

Frontend benchmark scores across evaluation dimensions

Grok 4.5 delivers the strongest balance of speed, cost, and functional depth. Its higher frontend score than Sol's reflects greater functional completeness despite the layout issues visible in the screenshot above, and it nearly matches Sol on the backend.

For me, Fable 5 is within a few percentage points of the real Basecamp frontend implementation, with very minor but detailed adjustments that a talented design engineer could finish.

The real Basecamp frontend showing its sample podcast project

The gap between Fable 5 and the other models is striking, especially given how heavily OpenAI1, Anthropic2, and SpaceXAI3 tout them for frontend work.

The final 10% of a project requires as much work as the first 90%.4 Each point on Basecamp Bench is harder-won than the last. Moving from eight to nine requires far more depth and complexity than moving from five to six.

Browse each frontend submission in the repository ↓

Fable 5

Sonnet 5

GPT-5.5

GPT-5.6 Sol

Grok 4.5

Run variability

Score variability across five Sonnet 5 and GPT-5.6 Sol runs

I ran Sonnet 5 and GPT-5.6 Sol five additional times on each track to measure run-to-run variation. Stronger runs turned more of the requested product into working, persistent behavior and handled API contracts and validation more carefully, while weaker runs often looked similarly complete at a glance but concealed inert controls, brittle state, and shallow backend behavior. The spread reveals each model's latent capability range and the risk of building from an unlucky draw.

Model / trackMedianObserved rangeMedian costBest of 5Δ vs. publishedSonnet 5 FE6.5856.178–6.922$13.726.922−0.060GPT-5.6 Sol FE6.0265.780–6.489$6.476.489+0.724Sonnet 5 BE7.4607.346–7.743$24.047.743+0.500GPT-5.6 Sol BE6.9956.675–7.313$11.407.313+0.003

Full report and other agents

I used the new GPT Sites feature to publish the evaluation report, where you can explore the methodology, dimension-level scores, charts, model-by-model commentary, and specific failure modes.

The GitHub repo contains the runner, prompts, rubrics, versioned contracts, reference material, baseline runs, evaluator reports, leaderboards, and provenance manifests needed to inspect or reproduce the benchmark.

Building the project with GPT-5.6 Sol consumed 1.8 billion tokens and $2,867.52 in model usage.

I attempted to test ZAI's GLM 5.2 using the Pi harness, but the model is unable to complete the benchmark as written.5 I also attempted to test Gemini Flash 3.5 in AGY, but experienced tool call failures across...