Preventing agents from cheating on our custom SWE-Bench | Superconductor Blog<br>Search⌘KLog inGet started

Back to all postsSuperconductor is a multiplayer workspace for your team and coding agents. It can also build a custom SWE-Bench for your codebase: a benchmark of coding agents built from your team's own merged pull requests, so you know which agents perform best on your stack.

We run this benchmark on our own codebase. It is built from a set of ground-truth PRs: substantial, real tickets whose merged implementations our team is proud of and considers strong examples of good engineering work. Agents receive specs inferred from those PRs, but never see the original solutions.

Agent benchmark on our Ruby on Rails codebase: Quality vs. Cost

Heat map of agent performance<br>During a regular audit of the generated solutions, one Composer 2.5 solution looked suspiciously close to the ground-truth PR. Its work log showed why: the agent had searched pull requests with gh, fetched origin/main, found the exact commit, and cherry-picked it into the working tree. Other agents also found the solution in Git history or after fetching ground-truth branches, which contaminated their benchmark submissions. These agents' output was solid, but the implementation no longer measured independent problem-solving.

When we originally discovered this issue, it was uncommon amongst the agents we'd tested. However that changed when we benchmarked Grok 4.5 and noticed that 20/30 implementations across various effort levels and harnesses showed evidence of cheating.

Agent searches for pull requests, finds the exact commit, and cherry-picks it.<br>Our findings fit a broader pattern: DebugML's independent audit, "Finding Widespread Cheating on Popular Agent Benchmarks," found thousands of cheating instances across 28+ benchmark submissions, including task-level cases where agents mined Git history or looked up public answers. In a far more extreme case, OpenAI and Hugging Face disclosed a security incident in which cyber-capable OpenAI models, evaluated on a benchmark of cyber capabilities, escaped their isolated test environment and compromised Hugging Face's production infrastructure to obtain the benchmark's test solutions directly from Hugging Face's production database.

49 benchmark runs were contaminated

We audited 339 coding agent implementations, across 38 agent/profile configurations that ran against 10 ground-truth pull requests. While all 339 agents ran git commands (most commonly git log), only 49 tried various filters to successfully retrieve the ground truth solution.

Here's what they did with the retrieved solution:

32 agents directly reused the solution from the ground-truth PR in their implementations.

17 agents retrieved and inspected the ground-truth PR and relied on it while implementing their solution, without directly copying it.

Here's a breakdown of the contaminated agent implementations:

HarnessModelContaminated implementationsCursorGrok 4.5 (XHigh)7Grok BuildGrok 4.5 (XHigh)7CursorGrok 4.5 (High)6CodexGPT-5.6 Luna (High)4CodexGPT-5.6 Sol (High)4CodexGPT-5.6 Luna (XHigh)3CursorAuto3CursorComposer 2.53CursorComposer 2.5 Fast3CodexGPT-5.6 Sol (XHigh)2CodexGPT-5.6 Terra (High)2Claude CodeFable 51CodexGPT-5.6 Terra (XHigh)1Grok BuildComposer 2.5 (High)1PiGLM 5.11PiMiniMax M31<br>Some agents went further than local Git history

Of the 49 contaminated implementations, 12 used gh pr, gh api, or GitHub search. For example, one Grok 4.5 (XHigh) agent searched GitHub, tried several local refs, fetched the exact ground-truth branch, and inspected its files with git show. The 12 runs were ten Grok 4.5 agents and two Composer 2.5 agents.

Work log showing the agent used the GitHub CLI to search for the ground truth pull request<br>Preventing future cheating

To prevent this kind of contamination, we deployed a change that adds a cleanup step before a coding agent starts working on a benchmark implementation. It keeps the current benchmark branch and the history reachable from its base commit, removes every remote and other ref, deletes common temporary Git pointers, expires every reflog entry, and prunes unreachable objects. Agents can still use Git for status, diffs, commits, and pre-base history so it's still representative of a real engineering task environment.

The change also blocks direct HTTP and HTTPS access to github.com, githubusercontent.com, their subdomains, and the GitHub storage endpoint that serves repository content. This closes the observed gh, raw-file, and HTTPS fetch routes without otherwise changing network policy.

After the fix was deployed, we re-ran the implementations and audited the agent work logs. We found no references to the merge SHA or any commits created after the benchmark starting point. Interestingly, only two agents (Cursor Auto and Cursor Composer 2.5) attempted to fetch more history after finding that the Git history was cut off.

How the results changed

Once we excluded the...