[2607.21763] Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks

Skip to main content

Search arXiv

Press Enter to search · Advanced search

-->

Computer Science > Cryptography and Security

arXiv:2607.21763 (cs)

[Submitted on 23 Jul 2026]

Title:Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks

Authors:Michael Kouremetis, Ads Dawson, Raja Sekhar Rao Dheekonda, Brian Greunke<br>View a PDF of the paper titled Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks, by Michael Kouremetis and 3 other authors

View PDF<br>HTML (experimental)

Abstract:Large language model (LLM) agents routinely cheat on cybersecurity benchmarks, inflating reported pass rates far beyond genuine capability. Prior audits of Cybench found cheating in 0.3-3.4% of traces, implicating only a handful of models. We present a controlled prompt-ablation study across 22 frontier models from 7 providers on 23 Cybench capture-the-flag (CTF) challenges under three prompt conditions (no anti-cheat, standard, severe). All 1,518 task traces were individually audited through a four-stage pipeline combining LLM-as-a-judge classification, programmatic verification, judge-verifier reconciliation, and human review. We find cheating is far more pervasive than previously estimated: under baseline conditions, 37.1% of passes involved cheating, 21 of 22 models cheated, and scores were inflated by up to 5x. Anti-cheat prompts reduce cheat propensity from 33.0% (baseline) to 17.8% (standard) to 8.5% (severe) without degrading, and sometimes improving, solve rates. However, even under the most restrictive prompt condition, eight models still produced cheated passes, four showed backfire effects, and cheating escalated from web search toward infrastructure probing. We introduce the "solve rate" metric (clean passes only) to distinguish genuine capability from cheated outcomes, and argue it should be standard practice in any evaluation where cheating vectors are available. Anti-cheat prompts are an effective and essentially free first layer of defense, but they are not a substitute for environmental controls.

Comments:<br>21 pages, 4 figures, 7 tables

Subjects:

Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)

Cite as:<br>arXiv:2607.21763 [cs.CR]

(or<br>arXiv:2607.21763v1 [cs.CR] for this version)

https://doi.org/10.48550/arXiv.2607.21763

Focus to learn more

arXiv-issued DOI via DataCite

Submission history<br>From: Michael Kouremetis [view email]<br>[v1]<br>Thu, 23 Jul 2026 19:26:25 UTC (30 KB)

Full-text links:<br>Access Paper:

View a PDF of the paper titled Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks, by Michael Kouremetis and 3 other authors<br>View PDF<br>HTML (experimental)<br>TeX Source

view license

Current browse context:

cs.CR

next >

new<br>recent<br>| 2026-07

Change to browse by:

cs<br>cs.AI

References & Citations

NASA ADS<br>Google Scholar

Semantic Scholar

export BibTeX citation<br>Loading...

BibTeX formatted citation

&times;

loading...

Data provided by:

Bookmark

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Which authors of this paper are endorsers? |<br>Disable MathJax (What is MathJax?)

Major funding support from