Do Skills Improve Coding Agent Accuracy? - OrcaBot Labs<br>Sign In

← All Labs<br>Agent skills were originally designed to allow developers to teach a general AI agent a specific process or domain expert’s workflow without bloating the agent's prompt or context window. Before agent skills, managing AI behavior was a messy process of swapping and chaining system prompts.

Almost immediately, developers such as Jesse Vincent 1 were developing skills to improve Claude Code and other harnesses' ability to handle general engineering tasks. Jesse's Superpowers 2 claimed to impose a professional methodology automatically; Get Shit Done (now Git Ship Done or simply GSD) 3 would turn an idea into durable specifications and phases; Oh My ClaudeCode 4 then promised to provide “zero learning curve” orchestration. Andrej Karpathy's tweets 5 about his coding workflow got quickly turned into another skill 6 and Addy Osmani released "Agent Skills" 7, production-grade engineering skills for AI coding agents.

None of them originally shipped with evidence, other than some demonstrations, that they actually improved end-to-end software-engineering performance. Superpowers came closest, but it tested workflow compliance, not whether the same model solved more coding tasks. And with the AI models and coding harnesses constantly improving, it was an open question whether any gains would survive the next model release.

So I decided to put these skills to the test using similar benchmarks to the ones that the frontier labs use when they release new models.

Results

SWE-bench Pro 8

731 problems, all complete, single run, model: Codex 5.5

#ArmResolve %Partial %Tokens/prob$/probΔ vs base1Oh My ClaudeCode54.99%76.1%2.09M$0.54+2.192Git Ship Done54.45%75.3%2.46M$0.60+1.643Agent Skills54.45%75.8%2.06M$0.51+1.644Superpowers-v554.17%75.8%1.72M$0.48+1.375Karpathy Skills53.08%74.7%1.23M$0.37+0.276baseline Codex 5.552.80%72.9%1.29M$0.38—<br>SWE-bench Pro contains long-horizon issues drawn from 11 actively maintained open-source repositories; a task may require substantial coordinated changes across several files, but the agent generally gets one issue and one final evaluation.

Note: While these benchmarks were running, Jesse Vincent released Superpowers v6. I will post an updated version of this benchmark soon.

SlopCodeBench 9

36 problems / 196 checkpoints, mean of 3 runs, model: Codex 5.5

#ArmStrictIsoCorePartial$/ckptErosion1baseline Codex 5.512.2 ± 0.425.768.441.71.320.582Git Ship Done11.9 ± 0.226.069.039.82.040.543Agent Skills11.7 ± 1.524.865.138.92.000.474Oh My ClaudeCode11.6 ± 2.325.963.642.61.870.525Superpowers-v511.4 ± 2.327.465.036.11.670.466Karpathy Skills11.1 ± 0.924.866.241.71.320.58<br>SlopCodeBench contains 36 synthetic, language-agnostic problems divided into 196 sequential checkpoints. The agent receives only an observable CLI or API contract, chooses its own architecture, and must keep modifying the code it previously wrote.

Verdict

For SWE-bench Pro, all the skills provide an improvement over baseline. This effect disappears with SlopCodeBench with all strict scores worse than baseline (although GSD gets higher Iso and Core pass rates).

With the exception of Karpathy Skills, each collection of skills costs more to run than baseline.

Potential Explanations

Many of these frameworks contain procedures intended for navigating and modifying established codebases. Those procedures have limited value during the initial greenfield checkpoint of SlopCodeBench and may impose a context or orchestration cost. SWE-bench Pro's hard part is finding the right 20 lines in a 500k-line unfamiliar repo and making a surgical edit whereas on SCBench the agent wrote the code itself so it will usually already be in context. Additionally all these instructions, routing decisions and procedural constraints add to the context. On a difficult repository task, that additional structure can focus the model. On a small task, it can compete with the actual specification for attention.

In SWE-bench Pro, the repository normally provides existing tests, related test patterns and observable regressions. A TDD or systematic-debugging workflow can use that evidence to localize the issue and protect surrounding functionality.<br>SlopCodeBench keeps its evaluator tests hidden. Agents can write tests only from the current external contract and examples. Skills push "reproduce the failure, write a failing test, then fix." but on SCBench there's no bug to reproduce.

So, do skills improve accuracy?

On the evidence: sometimes, and it depends on the task.

On SWE-bench Pro, yes — uniformly. Every skill collection helped.

On SlopCodeBench, no — every skill collection actively hurt.

A caveat worth stating plainly: SlopCodeBench ran at n=3 seeds and the spread on several arms (±2.3) is wider than the gaps between them.

Methodology

Try it yourself in OrcaBot

There is a template for running SlopCodeBench within OrcaBot which acts as the benchmark orchestrator for you...