SolarBench
SolarBench<br>Can AI agents maintain industrial infrastructure?
The first benchmark for industrial operations.<br>The physical economy is $5 trillion of US GDP and is built on complex long horizon and complex industrial operations. SolarBench measures whether an agent can handle that real-world complexity by evaluating it on a simulated on-call solar operations desk.
Live · SolarBench, the gameFable 5 scored 96% on an easier, single-day version. Can you do better?<br>Beat the AI →
Introduction<br>We believe the messy physical world is the next horizon for agents. SolarBench measures whether a model can run an industrial operations desk.<br>SolarBench places a model in the seat of a remote operations desk responsible for a portfolio of solar sites. The simulated desk has the tools of the industry's back office: an alarm feed, per-site telemetry, work orders, parts inventory, and an inbox of owners and technicians. The model has to figure out what is actually broken and get people and parts into the field.<br>We chose solar because we believe it is a happy medium for a complete test: running a portfolio blends operational, financial, and human judgment over a long horizon. Remote operation desks manage dozens of sites with their own specifications and history, real revenue on the line, and a rotating cast of technicians, owners, and managers.<br>The scenarios are drawn from hundreds of hours with the people who actually run solar, reconstructed as a live operations desk.
How It Works<br>Operations Desk Diagram<br>the world<br>Telemetry<br>Revenue meters<br>Documents<br>Inbox<br>Techs & parts
world stateseeded, event-quantized week clock, one record every surface writes into
↑ tool calls: read any surface (free), act on the world (priced)↓ observations, consequences, the bill<br>Agent seatmodel + harness, sandboxed, queries the world only through tools, files the Sunday report
The solar world is very messy: alarms are often not reliable on their own and some real faults never raise an alarm (e.g. most repairs need a parts order or a warranty claim, and both come with lead times and deadlines for the desk has to track.) Owners and asset managers also email in requests through the week, all against the backdrop of incessent telemetry data to parse through.<br>We evaluate the model in a weeklong simulation with a specific, expert-drawn, long-horizon issue it needs to resolve. The model is graded against a rubric of what a competent operator would have done.
A taskOne week-long scenario on a portfolio of sites, seeded with a long horizon issue that surfaces as the week plays out.<br>A passA run passes only if every aspect of the surfaced issue is properly resolved by the end of the week. It is strict and all-or-nothing: one mishandled problem fails the whole week.<br>The launch setEight tasks, eleven models, ten runs each: 880 graded weeks in total.<br>The gradeEach run's end state is checked against an authored rubric of what a competent operator would have done, rather than a fixed transcript to imitate.<br>Every figure below is built from those 880 runs.
Model performance<br>We measured the profit of each model, normalized such that $0 is a week with nobody at the desk: no production saved, no money spent.
Portfolio Profit<br>★Oracle +$8,483<br>1Claude Fable 5 +$6,822<br>2GPT-5.6 Sol+$6,591<br>3GLM 5.2+$6,575<br>4Kimi K3+$4,714<br>5Grok 4.5+$4,201<br>6Gemini 3.5 Flash+$3,765<br>7GPT-5.6 Terra+$3,316<br>8Muse Spark 1.1+$3,281<br>9Kimi K2.7 Code+$3,242<br>10Claude Sonnet 5+$3,239<br>11Gemini 3.1 Pro+$2,948
Median run per model.
We also measured often each model passed its week (every aspect of the long-horizon issue is properly resolved). The strongest model manages this about half the time.
0%25%50%75%100%Claude Fable 5: 53.8% pass rate, +/- 1 SE 5.6% (Wilson 95% CI 42.9% to 64.3%, 80 runs)Claude Fable 553.8%±5.6%Muse Spark 1.1: 42.5% pass rate, +/- 1 SE 5.5% (Wilson 95% CI 32.3% to 53.4%, 80 runs)Muse Spark 1.142.5%±5.5%Grok 4.5: 38.8% pass rate, +/- 1 SE 5.4% (Wilson 95% CI 28.8% to 49.7%, 80 runs)Grok 4.538.8%±5.4%GPT-5.6 Sol: 33.8% pass rate, +/- 1 SE 5.3% (Wilson 95% CI 24.4% to 44.6%, 80 runs)GPT-5.6 Sol33.8%±5.3%Gemini 3.1 Pro: 28.8% pass rate, +/- 1 SE 5.1% (Wilson 95% CI 20% to 39.5%, 80 runs)Gemini 3.1 Pro28.8%±5.1%Kimi K3: 26.3% pass rate, +/- 1 SE 4.9% (Wilson 95% CI 17.9% to 36.8%, 80 runs)Kimi K326.3%±4.9%GLM 5.2: 23.8% pass rate, +/- 1 SE 4.8% (Wilson 95% CI 15.8% to 34.1%, 80 runs)GLM 5.223.8%±4.8%Claude Sonnet 5: 12.5% pass rate, +/- 1 SE 3.7% (Wilson 95% CI 6.9% to 21.5%, 80 runs)Claude Sonnet 512.5%±3.7%Gemini 3.5 Flash: 11.3% pass rate, +/- 1 SE 3.5% (Wilson 95% CI 6% to 20%, 80 runs)Gemini 3.5 Flash11.3%±3.5%GPT-5.6 Terra: 6.3% pass rate, +/- 1 SE 2.7% (Wilson 95% CI 2.7% to 13.8%, 80 runs)GPT-5.6 Terra6.3%±2.7%Kimi K2.7 Code: 5% pass rate, +/- 1 SE 2.4% (Wilson 95% CI 2% to 12.2%, 80 runs)Kimi K2.7 Code5%±2.4%Macro-average pass rate over the eight launch tasks, 10 runs per model per task. A run passes only if all surfaced issues during the week are properly...