A coding agent fixed only 14% of vulnerable advisories. Bomly MCP set a 98% floor. — Bomly

Give a coding agent the vulnerable-dependency list and fix context over MCP, and remediating a large project stops being a gamble. That is the result of a study we ran with two leading agents — Claude Code on Opus 4.8 and Codex CLI on GPT-5.5 — on a 13-module Maven project with about 300 dependencies. Every run with the Bomly MCP server connected finished at 98% complete or above, where complete means the share of fixable advisories actually resolved. What the server fixed differed by agent: Claude Code's bare runs swung between 14% and 98%, and claimed more fixes that did not hold up; Codex CLI's bare runs were already at 93–100%, but took almost twice as long. On three smaller apps, the agents did well on their own; we report that too, because it marks where the server earns its place and where it doesn't.

The large-project result in four numbers (five runs per agent and condition):

ResultBareBomly MCPClaude Code — lowest completeness14%98%Claude Code — runs with wrong claims4 of 52 of 5Codex CLI — lowest completeness93%100%Codex CLI — average runtime~13 min7.5 min

Note

Everything is public: fixtures, exact prompts, raw transcripts, scoring code, and every number in this post is in bomly-dev/bomly-agent-study. The long-form writeup with all tables lives there too, in REPORT.md.

What we ran

Two agents, pinned and recorded per run:

VersionModelReasoningClaude Code2.1.201claude-sonnet-5 (three smaller fixtures) · claude-opus-4-8 (large fixture)highCodex CLI0.142.5gpt-5.5medium

Each agent ran in two conditions on each fixture:

bare — the agent, its own tools, and open network access.

mcp — the same setup plus a running Bomly MCP server (bomly mcp serve), which can return the dependency graph, the vulnerable-package list, and fix context.

The large fixture got five runs per agent and condition. The three smaller fixtures got one: after the first round came back at or near the ceiling, we dropped their remaining planned rounds and moved that budget to the large fixture.

The task prompt is identical in both conditions: find and fix the vulnerable dependencies, keep the build green. The instruction files differ by one equal-length line disclosing which tools are available. Each run starts from a fresh copy of the fixture with the answer key and scoring code stripped out, inside a pinned Docker image, and is scored mechanically afterward against a frozen ground truth: the full set of confirmed-vulnerable packages, not a hand-picked subset. METHODOLOGY.md has the details, including how the study design changed along the way and why.

Three of the four fixtures are real applications vendored at old, genuinely vulnerable tags: CTFd 3.7.7 (Python), Dependency-Track 4.10.0 (single-module Maven), and — the large one — Internet2 Grouper 4.x: a 13-module Maven build with roughly 300 resolved dependencies, 21 confirmed-vulnerable packages, and 56 fixable advisories. The fourth is a small npm app built for the study. No vulnerability anywhere is invented: every advisory is a real, published one against a real package version.

Limitations, first

N=5 per cell on one large-project fixture, and N=1 per cell on the three smaller fixtures, so these are observations from our setup, not general claims. On the large fixture the vulnerable surface comes from Bomly's own resolution (the free independent scanners we used elsewhere cannot resolve transitive dependencies for multi-module Maven, and the commercial tools that can were not available to us), with hand-verified spot checks — and whether a fix actually landed is checked from the build and the diff, independently of Bomly. The two agents ran different models with different settings, so we compare each agent only to itself. And because some runs varied a lot, every chart shows every run, not means. The full list is in LIMITATIONS.md.

On the smaller apps, there was little room to improve

This was supposed to be a difficulty ladder. It turned out to be a ceiling: on the npm, Python, and single-module Maven fixtures, five of the six bare runs finished at 99–100%. The exception was one Claude Code run on the Python fixture, which stopped at 56% and broke the shared build. With one run per cell here, we do not read the bare-versus-MCP differences as evidence of an effect — the useful result is that capable agents had little completeness headroom on small projects.

Small projects are within reach of a capable agent on its own: read the manifests, run npm audit or pip-audit where they exist, bump versions. In our setup the server added no completeness there, and we would rather say that plainly than stretch a small difference. Two side notes survived even here: three runs — two bare, one MCP — finished "complete" with the shared build broken (the dagger marks), and Codex CLI handled the one genuinely unfixable advisory on the Maven fixture better with the server connected — it...