Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal Help? - Charles AZAM

fr<br>fr

Blog<br>TL;DR: I gave Claude Fable 5 and GPT-5.6 Sol the same unpublished NP-hard<br>optimization problem, with and without their native /goal mode. Fable 5 is an absolute beast;<br>/goal is not a game changer.

Context: This is an operations research problem originally submitted to students at a<br>hackathon. I spent a week years<br>ago writing C++ to solve it, so I have a useful human baseline.

Fable 5 was an absolute beast on this benchmark. It produced the best solution overall, and its<br>consistency is unlike anything I have seen from a model on this problem. This is pure raw<br>intelligence. Incredible.

The other result is that /goal is not a generic “try harder” switch. It changes the control<br>loop and the search path. Sometimes that finds a better basin. Sometimes it gives a bad idea<br>more time to mature.

All code, prompts, result tables, exclusions, and trajectory notes are in<br>CLIArena. This is a follow-up to my<br>first article about this benchmark.

The problem

KIRO is a fiber-network design problem I worked on as an engineering student in 2018. Given<br>directed distance matrices for Grenoble, Nice, and Paris, the solver has to connect distribution<br>points and terminals using loops and short chains while respecting several structural<br>constraints. The objective is total cable length. Lower is better.

A valid network consists of redundant loops rooted at distribution hubs, with short branches<br>hanging from towers on those loops. Every tower must appear exactly once, and reversing a cable<br>segment can change its cost.

How large is the search space?

There is no single closed-form count because a solution can use any number of loops, variable<br>loop sizes, and differently anchored and ordered branches. But Paris alone gives a useful lower<br>bound.

Even if we ignore ordering and branches and only assign each of the 532 terminals to one of<br>11 distribution hubs, there are 11^532 possible assignments.

A stronger lower bound comes from one deliberately restricted family of valid solutions:<br>exactly 19 loops of 28 terminals each, with no branches. This covers all 532 terminals because<br>19 x 28 = 532, while staying below the 30-terminal limit for a loop. Order the 532 terminals,<br>split that ordering into 19 consecutive groups, divide by 19! because the set of loops is<br>unordered, and choose one of the 11 hubs for each loop:

(532! / 19!) x 11^19 ~= 10^1223

What I tested

The primary experiment was intentionally narrow:

SettingValueModelsClaude Fable 5, Opus 4.8, Sonnet 5; GPT-5.6 Sol, Terra, LunaModesPlain; native /goalOptimization budget30 minutesOuter agent timeout1,900 secondsReasoningMaximum available setting for every modelExecutionHarbor 0.1.43, Docker, subscription authentication<br>Results

Before concentrating repetitions on the flagship pair, I ran one matched 30-minute no-hint<br>pair for every model in the sweep. For Fable and Sol, the chart uses Pair 1 from the replicated<br>headline set; the other four models have one pair each.

I then repeated the flagship comparison until I had three matched runs for Fable 5 and three<br>for Sol.

ModelRunPlain/goalGoal minus plainFable 5132,19731,934 -263Fable 5232,51632,324 -192Fable 5332,446 35,178+2,732GPT-5.6 Sol133,581 39,371+5,790GPT-5.6 Sol235,53932,703 -2,836GPT-5.6 Sol333,66333,313 -350<br>Negative means /goal was better. Goal won four of six trials, so win rate alone makes the<br>feature look useful. The means tell the other half:

ModelPlain mean/goal meanMean effectMedian effectFable 532,386 33,145+759 worse-192 betterGPT-5.6 Sol34,261 35,129+868 worse-350 better<br>Both models usually got a small benefit and occasionally suffered a large regression. That is<br>why /goal won most runs but made both means worse.

Fable was also clearly stronger. Its plain mean beat Sol’s by 1,875 points, and its goal mean<br>beat Sol’s by 1,984. More importantly, Fable plain stayed inside a tiny 319-point range while<br>Sol plain spanned 1,958 points. Fable goal produced the best clean score, 31,934; Fable plain<br>was the safest configuration.

Deep dive into the goal command

The same command hides two different systems

Claude Code and Codex both expose /goal, but the implementations are fundamentally different.

Claude Code: a separate evaluator

Claude Code implements /goal as a session-scoped Stop hook. After each main-model turn, a<br>small evaluator model, Haiku by default, reads the condition and conversation. It returns yes<br>or no with a reason. A no starts another turn; a yes clears the goal.

The evaluator cannot use tools or inspect files. It can only judge evidence that appeared in<br>the transcript. That can catch an early exit, but it cannot know whether another ten million<br>solver iterations are worthwhile. Anthropic’s goal documentation

Keep in mind that claude code is not open source, so we rely solely on what Anthropic tells us.

Codex: persisted state and lifecycle tools

I also read the source for the benchmarked...