CrucibleBench — Old Worlds for New Agents
Skip to main content
The idea
Lateral thinking with withered technology
Nintendo's Gunpei Yokoi used the phrase to describe a design philosophy: take mature, inexpensive,<br>well-understood technology and use it in a new way. CrucibleBench applies it to AI evaluation.
Instead of photorealistic simulation or browser automation, we start with a MUD : a<br>multi-user dungeon, the persistent text worlds of the early internet. Its constraints are the point: a<br>limited command space makes hallucinated actions detectable, NPCs with trust and suspicion state give<br>explicit social feedback, and within-run persistence means items taken stay taken and trust earned stays<br>earned.
We did not choose a MUD because it is charming. We chose it because its constraints make behavior<br>measurable.
Why a MUD
Old constraints solve modern measurement problems
Static benchmarks measure what models know in isolation. They do not measure how models<br>behave where trust must be earned, information is gated by relationships, and blunt questioning raises<br>suspicion.
01
An enumerable action space
7 command types, 12 rooms, 14 items. Hallucinated actions and wrong-room interactions are detectable,<br>and action efficiency is measurable.
02
Explicit social feedback
4 NPCs carry trust and suspicion state (0–100) that moves in response to dialogue: feedback a model<br>can adapt to within a run, or fail to.
03
Within-run persistence
Items taken stay taken; trust earned stays earned. Every run leaves a complete, replayable transcript<br>of exploration and planning.
What $99 bought
The central finding is about measurement, not rankings
A single LLM-judge component inside the scoring stack reordered the leaderboard by up to six<br>positions, while every aggregate reliability statistic stayed silent. We report every result under two<br>scoring configurations and treat the divergence as the paper's most generalizable finding.
Judge ablation reorders the top of the board
Two of four scored dimensions route through a dialogue classifier whose per-model agreement with an<br>independent judge spans 21.7% to 84.8% , instability the aggregate κ = 0.04<br>never reveals. Removing the classifier-dependent dimensions shifts six rankings beyond scenario-sampling<br>noise (90% paired block bootstrap).
The largest mover shares a model family with the classifier. Benchmarks that use LLM judges should report<br>per-subject agreement and ranking stability under judge ablation, not aggregate reliability alone.
Robust rank shifts under classifier-minimized scoring
Model<br>Full<br>CM
Claude Sonnet 4.6<br>#4<br>#1<br>▲ 3
DeepSeek R1<br>#7<br>#2<br>▲ 5
Grok 4<br>#12<br>#10<br>▲ 2
GPT-5.4<br>#1<br>#5<br>▼ 4
Gemini 3.1 Pro<br>#3<br>#9<br>▼ 6
Mistral Large 3<br>#10<br>#12<br>▼ 2
Model<br>Classifier-min.<br>Full score<br>Success<br>$ / run
Claude Sonnet 4.6<br>4.04<br>3.89<br>24%<br>$0.125
DeepSeek R1<br>4.00<br>3.85<br>22%<br>$0.119
Claude Opus 4.6<br>3.93<br>3.93<br>30%<br>$0.205
GPT-5.2<br>3.91<br>3.88<br>38%<br>$0.113
GPT-5.4<br>3.88<br>4.07<br>68%<br>$0.060
Qwen 3.5 397B<br>3.81<br>3.81<br>30%<br>$0.017
Claude Haiku 4.5<br>3.80<br>3.88<br>34%<br>$0.039
GPT-5.3 Chat<br>3.73<br>3.72<br>40%<br>$0.095
Gemini 3.1 Pro<br>3.71<br>3.91<br>48%<br>$0.339
Grok 4<br>3.61<br>3.48<br>32%<br>$0.834
DeepSeek V3.2<br>3.60<br>3.61<br>24%<br>$0.008
Mistral Large 3<br>3.44<br>3.69<br>40%<br>$0.017
OLMo 3.1 32B<br>2.01<br>1.93<br>4%<br>$0.005
Mean scores on a 1–5 rubric scale, sorted by classifier-minimized subtotal. 50 runs per<br>model: 5 seeds × 2 objectives × 5 repetitions, temperature 0.3, billing-verified via OpenRouter. Rankings<br>are exploratory; confidence intervals overlap substantially among the top eight. Full protocol, CIs, and<br>statistics in the whitepaper.
Behavioral failure modes
Failures you can read in the transcript
Three failure modes, each detected algorithmically from state-machine telemetry, with no judge<br>involved. Dialogue looping is the dominant mode for every model tested, frontier included.
Dialogue looping 14–66% of frontier runs
Eight or more talk commands at a single NPC in one run. The agent repeats a failed conversational<br>approach instead of adapting: the persistent-world cousin of a support agent repeating itself.
Wrong-room interaction severe in floor model
A talk command answered by "no one here." Reveals lost world-state tracking, analogous to calling an<br>API that is not in scope. Grok 4 was the only frontier model with meaningful incidence (12%).
Exploration paralysis selective, floor-dominant
Two or fewer rooms across twenty-plus turns, or five consecutive look commands. Information gathering<br>that never becomes goal-directed action.
middleham · gain_watch_trust<br>olmo 3.1 32b · seed 20260399
turn 04> talk guard "Hello, I'm<br>new to Middleham. Could you tell me about The Watch?"
No one by that name is here.
turn 05> talk guard "Excuse me, do<br>you know where I can find an officer of The Watch?"
No one by that name is here.
turn 17> talk street_crystal<br>"Hello, can you help me?"
No one by that name is here.
turn 36> talk captain ser alarik
[15 consecutive turns at the captain · objective never completed]
Verbatim from...