Who benchmarks the benchmark? Auditing an agentic gym
Agent failures are often environment<br>failures disguised as model failures.
To detangle the failure modes, we build benchmarks (also called gyms)<br>that control the environment so we can better study and improve the<br>model in isolation.
In this post, I seek a method to score benchmarks themselves.
Agents train at the Gym
Gyms are typically made up of these 7 components:
Task specification: the user prompt and initial context
Agent policy: the system prompt, finetuning, or other behavior<br>nudging
Planning & reasoning: the model<br>itself
Tool contract: the list of tools available, their descriptions, and<br>their params
Tool implementation: how executing each tool affects the state
Environment state: the simulation of the real world
Verifier: assertions to judge the outcome of the gym scenario<br>(pass/fail)
That’s a lot of points of failure! However, only #3 measures the<br>model itself.
When you investigate a failing gym scenario, these are common<br>symptoms you may witness:
“Won’t follow instructions”
“Flaky: passes sometimes”
“Hits a hard ceiling”
“Just not smart enough”
The problem is, it’s not always easy to attribute causes to the<br>symptom. Click around to see what I’m talking about:
Task specificationambiguous prose<br>Agent policyoverconstrained rules<br>Planning & reasoningactual model error<br>Tool contractdocs ≠ behavior<br>Tool implementationpartial side effects<br>Environment stateundefined / drifting<br>Verifierunsatisfiable checks
“Won’t follow instructions”<br>“Flaky: passes sometimes”<br>“Hits a hard ceiling”<br>“Just not smart enough”
ALL OF THE ABOVE GET BLAMED ON THE MODEL
click a node
Gyms are hard to do right
The Agentic Benchmark<br>Checklist assessed 10 widely used agentic benchmarks and found 7<br>violating task validity and 7 violating outcome validity. A separate<br>audit broke<br>8 benchmarks without solving a single task, most of them to a<br>near-perfect score. SciCode-Verified found<br>262 defects across 63 of its 64 problems, 192 of them rejecting correct<br>solutions. Famously, OpenAI stopped reporting SWE-bench Verified after<br>finding that 59.4%<br>of the problems its models failed were themselves broken, with 35.5%<br>carrying tests so narrow they reject functionally correct<br>submissions.
Let’s zoom in on one. The EnterpriseOps Gym<br>measures an agent’s ability to follow complex instructions to operate a<br>set of real-world MCP tools. It’s 1150 expert-curated tasks across 8<br>domains, 7 to 30 steps each, running against live containerized MCP<br>servers with real state. The results are in the chart below. Fable 5<br>scores 52% on the “Teams” domain.
52% reads like the goal to beat, but you may be surprised to learn<br>that I took the teams/oracle split (61 tasks)<br>from 26.2% to 100% with gpt-5.6-luna by fixing various<br>issues in the gym itself.
Here’s the breakdown:
26.2%→100%
73.7 points recovered across six defect classes
points recovered<br>cumulative
80%
100%
+31.1<br>+19.7<br>+9.8<br>+8.2<br>+3.3<br>+1.6
Contradictions in the environment<br>Flaky verifiers<br>Overspecified policy<br>Impossible verifiers<br>Mismatched tool sets<br>Undefined clock
Contradictions in the environment+31.1
Flaky verifiers+19.7
Overspecified policy+9.8
80%
Impossible verifiers+8.2
Mismatched tool sets+3.3
Undefined clock+1.6
1.<br>Contradictions in the environment: +31.1 points, ~19 tasks
Contradictions are the largest single source of failure. Well, there<br>are two types:
(1) Misleading observations. For example,<br>create_virtual_event_townhall commits the row and<br>then returns None, which results in an error, even<br>though the write succeeded:
Failed to create townhall: schemas.virtual_event_townhall.VirtualEventTownhallResponse()<br>argument after ** must be a mapping, not NoneType<br>The confusing error message causes the agent to retry, and strict<br>verifiers fail this scenario. This one was reported<br>in March with the affected task ids and acknowledged by the<br>maintainers, and it is still open.
(2) Misleading descriptions. For example,<br>add_channel_member documents that the operation is<br>“allowed only for channels with a membershipType value of private or<br>shared”. That’s true of real Teams and false of this server, which<br>accepts standard channels and writes the row. An agent that believed the<br>documentation correctly skipped the call and was graded wrong.
This second case is the more damaging one, because it rewards<br>agents that don’t follow instructions . The benchmark’s<br>own system prompt orders the agent to “never infer” and to “abort with a<br>reason”, and then the environment penalizes exactly that compliance.
One more example, just for good measure: the tab tools ship an<br>examples value pointing at app id<br>06805b9e-77e3-4b93-ac81-525eb87513b8, and the server<br>rejects it:
Teams app '06805b9e-77e3-4b93-ac81-525eb87513b8' not found in organization app catalog.<br>Five tasks need a tab app id and have no app-listing tool in their<br>oracle set, so the documentation is the only...