ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?

Leaderboard

Successful exploits per model under a two-hour timeout, broken down by domain.<br>-->

survives mitigations

blocked by mitigations

bar length = exploits without mitigations · hover for exact counts

Success is the number of instances exploited via the intended vulnerability, split by domain:<br>userspace, browser V8, and Linux kernel. Evaluated under trusted access programs designed for security<br>research; evaluation notes are shown with each submission. Cost, output tokens, time, and LLM calls<br>(under Avg / task) are per-task averages over the full benchmark; hover for the averages over<br>successful exploits.

Key Takeaways

Autonomous exploitation is no longer hypothetical

Frontier agents can take a bug report and a crashing input, reason about memory layouts, chain<br>multiple attack primitives, and produce fully working exploits — work that traditionally required<br>deep human expertise and significant time.

Standard defenses help, but don't fully stop attacks

With ASLR, stack canaries, and the V8 heap sandbox enabled, successes dropped substantially but<br>didn't hit zero. Agents found bypasses such as partial-pointer overwrites, known sandbox escapes, and<br>kernel tricks like overwriting modprobe_path.

This is inherently dual-use

Automated exploit generation can accelerate severity triage and validate mitigations, but the same<br>capability lowers the barrier for offensive misuse. We believe the responsible path is to measure<br>these capabilities rigorously and openly.

What is ExploitGym?

Most existing cybersecurity benchmarks for AI focus on finding bugs, writing patches, or solving CTF<br>puzzles. Our earlier benchmark, CyberGym, focuses on real-world vulnerability<br>analysis: given a description and a codebase, agents must generate proof-of-concept inputs that trigger<br>a bug. That's an important step, but it stops short of the next question: can an agent turn a<br>known bug into a real attack?

ExploitGym fills that gap. Each of its 869 tasks provides the agent with: the vulnerable<br>source code with build instructions, a proof-of-vulnerability (PoV) input that triggers the bug, and a<br>containerized runtime environment. The agent's task is to transform that PoV into a working exploit that<br>achieves unauthorized code execution, concretely, retrieving a secret flag that is inaccessible through<br>any legitimate interface.

Figure. Each task hands the agent vulnerable source code, a proof-of-vulnerability<br>input, and a containerized runtime; the agent must produce an exploit that reads a secret flag. The 869<br>tasks span three domains, each with toggleable mitigations.

869<br>total tasks

502<br>userspace

181<br>V8 engine

186<br>Linux kernel

Userspace programs (502 instances) cover widely used C/C++ projects like FFmpeg and<br>OpenSSL, sourced from OSS-Fuzz and OSV. V8 browser engine tasks (181 instances) target<br>JavaScript engine bugs in Chromium. Linux kernel tasks (186 instances) require<br>full-privilege escalation inside a virtual machine. In addition to validating code execution through<br>flag capture, an agent-as-a-judge verifies that each exploit actually targets the provided<br>vulnerability.

The Interesting Bits

Agents go off-script and find new bugs

Across models, agents frequently achieved code execution through a vulnerability other<br>than the one provided. GPT-5.5 captured flags in 210 instances but only 120 used the intended bug;<br>Claude Mythos Preview captured 226 but only 157 targeted the right flaw. In some cases agents<br>pivoted to an adjacent code path with weaker validation; in others they concluded the given bug<br>wasn't exploitable and searched for new attack surfaces — by auditing source or even dynamic<br>fuzzing. A remarkable display of autonomous security reasoning.

Different models find different exploits

Claude Mythos Preview and GPT-5.5 dominate in total count, but their success sets diverge: 56<br>targets are solved exclusively by Claude Mythos Preview and 26 exclusively by GPT-5.5, with only 91<br>shared. The remaining models contribute another 61 successes, four of them unique. This suggests<br>the models rely on qualitatively different exploitation strategies — and that an ensemble approach<br>could substantially expand coverage.

More budget helps — but only for the best models

Extending the budget from two to six hours, Claude Mythos Preview kept climbing from 127 to 204<br>successful exploits with no clear plateau, while Claude Opus 4.6 flatlined around 15 within the<br>first 30 minutes. Frontier models are capable of sustained, multi-stage reasoning that can crack<br>harder problems given enough runway — meaning the two-hour budget likely undercounts what the<br>strongest agents can do.

Example: From a 5-Line Crash to Full Code Execution in V8

GPT-5.4 was given a five-line PoV that triggers an assertion in Maglev, V8's mid-tier JIT compiler,<br>reported by ClusterFuzz after GPT-5.4's knowledge cutoff. On the release build, the PoV just<br>throws a benign...