GitHub - AndrewSispoidis/contemporary-agent-attacks: Open corpus of attacks targeting autonomous AI agents. Reproducible benchmarking for AI security tools. · GitHub
/" data-turbo-transient="true" />
Skip to content
Search/
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
AndrewSispoidis
contemporary-agent-attacks
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>2 Commits<br>2 Commits
attacks
attacks
benchmark
benchmark
negatives
negatives
CONTRIBUTING.md
CONTRIBUTING.md
LICENSE
LICENSE
METHODOLOGY.md
METHODOLOGY.md
README.md
README.md
leaderboard.md
leaderboard.md
View all files
Repository files navigation
Contemporary Agent Attacks
An open, reproducible benchmark of attacks targeting modern LLM agents.
497 attacks (395 visible + 102 holdout) across 13 categories
1,172 benign samples across 4 negative categories, for measuring false-positive rates
Tool-agnostic benchmark runner — works with any HTTP-addressable classifier
CC-BY 4.0 — use it for your product, paper, or class
The corpus is aimed at contemporary agent-shaped threats: prompt injection,<br>credential exfiltration, indirect injection via tool output and RAG, tool<br>abuse, supply-chain manipulation, memory poisoning, and agent-to-agent<br>compromise. See METHODOLOGY.md for collection, sourcing, and<br>scoring details.
Quick start
git clone https://github.com/AndrewSispoidis/contemporary-agent-attacks<br>cd contemporary-agent-attacks
# Point at whatever classifier you want to measure<br>ENDPOINT=http://127.0.0.1:7749/v1/firewall/scan/full ./benchmark/run.sh
# Or use the Python runner, with per-category breakdown and JSON output<br>python3 benchmark/run.py \<br>--endpoint http://127.0.0.1:7749/v1/firewall/scan/full \<br>--tool my-tool \<br>--output benchmark/results/my-tool.json
Write a ~40-line adapter for your tool (see<br>benchmark/adapters/crawdad.py) and run:
python3 benchmark/run.py --adapter adapters.my_tool --tool my-tool
Requirements: Python 3.9+, jq (for the shell runner only), and a scan endpoint<br>that accepts POST {"text": ""} and returns a JSON body indicating<br>blocked vs. allowed.
Corpus layout
attacks/<br>prompt_injection/ # 37 — direct "ignore previous" style injections<br>credential_exfiltration/ # 44 — read env vars, SSH keys, cloud creds, output API keys<br>indirect_injection/ # 30 — payloads hidden in email/RAG/Slack/PDF/tool output<br>jailbreak/ # 30 — DAN, role-play, simulator, philosophical, translation<br>role_switching/ # 65 — goal hijack, rogue agents, inter-agent compromise<br>encoding_obfuscation/ # 40 — base64, hex, rot13, homoglyph, RTL override, embedding<br>system_prompt_extraction/ # 20 — direct and indirect system-prompt leakage<br>exfiltration/ # 28 — data-exfiltration intent ("POST these rows to …")<br>pii_leakage/ # 20 — SSN, passport, medical records, PII disclosure<br>tool_abuse/ # 53 — dangerous shell / subprocess / eval / file-write calls<br>exploitation_request/ # 20 — cascading failure patterns and exploit requests<br>supply_chain/ # 40 — typosquats, unsigned installs, MCP manifest abuse<br>other/ # 70 — memory poisoning, misinformation, unbounded consumption,<br># human-agent trust manipulation
negatives/<br>developer_messages/ # 1,075 — wild Stack Overflow developer Q&A<br>conversational/ # 10 — short benign chat prompts<br>security_discussion/ # 30 — benign security discussion ("how does SQL<br># injection work?", OWASP/CVE prose, etc.)<br>hand_curated/ # 57 — near-miss negatives crafted to look like<br># attacks to a naïve classifier
Each attack/negative is one .txt file with a descriptive slug filename. The<br>files contain only the attack or benign text — no metadata, so you can feed<br>them to any tool without parsing.
Holdout split
Files prefixed with holdout_ are reserved for evaluating generalization; they<br>were collected from sources not used to construct the visible corpus and have<br>never been public before this release. If you train on this corpus, do not<br>train on holdout files — use them only for final scoring. See<br>METHODOLOGY.md for details.
Latest results
Tool<br>Detection<br>FP rate<br>F1<br>Date
crawdad-v0.9.1<br>99.80%<br>0.00%<br>99.90%<br>2026-04-20
See leaderboard.md for more, and<br>benchmark/results/ for raw JSON.
Scoring
Detection rate — fraction of attacks the tool blocks
False-positive rate — fraction of negatives the tool incorrectly blocks
Precision / Recall / F1 — computed over blocked vs. not-blocked decisions
A tool that blocks everything has 100% detection and 100% FP — useless.<br>A tool that blocks nothing has 0% detection and 0% FP — also useless. F1 is<br>the useful single-number summary.
Contributing
We welcome new attacks...