Codex Skill Benchmark — Codex How To
Six controlled GPT-5.6-sol runs<br>Do Codex skills<br>save tokens? It depends.<br>The same engineering-loop skill lost on a small fix and won on a medium build. Explore the result, inspect the evidence, then run your own replication.<br>Explore the benchmarkReplicate it ↗
6 controlled runs<br>6/6 accepted<br>2 task sizes<br>0 human corrections
Quality gates came first. Token use was compared only after every variant passed.
Task-size boundary<br>One skill. Opposite outcomes.
Small, bounded fixBackend fixMedium implementation2048 build
Medium implementation<br>Dependency-free 2048
All passed<br>Four browser-game files, ten engine tests, syntax checks, and a post-run evaluator.<br>No skill
828,446<br>Full v0.2
553,179<br>Lean v0.4fewest tokens
380,767
VariantTimeRetriesAccepted<br>No repository skill350s1Yes<br>Engineering loop v0.2.0257s1Yes<br>Lean engineering loop v0.4.0247s0Yes
−54.0%lean skill vs control<br>The lean loop used 54.0% fewer tokens than the control.<br>It also used 31.2% fewer tokens than v0.2.0 and finished 29.4% faster than the control. Every variant still passed.<br>Read the 2048 measurement ↗
What was held constant<br>Evidence before conclusions.
Same model, reasoning effort, starting commit, task contract, sandbox, and acceptance criteria. Only repository-skill routing changed.
01Quality first<br>Acceptance, required checks, and evidence completeness were primary. A cheaper failed run would not win.
02Three variants<br>No repository skill, the original v0.2.0 loop, and the lean v0.4.0 loop started from equivalent fresh copies.
03Reported usage<br>Token totals are Codex CLI input plus output tokens. Cached input is already included and was not counted twice.
Read this before sharing<br>This is a boundary to test, not a universal claim.
Two seeded tasks and six total runs are not a population.<br>Global personal skills remained visible to every run.<br>Run order was fixed, not randomized.<br>Live browser behavior could not be verified in the sandbox.<br>Independent replications may—and should—disagree.
Make the evidence better<br>Run the protocol on your task.<br>Fork the fixture, hold the environment constant, report every result, and publish negative findings too.
Join the replicationFork an edition ↗