ShellRisk-Bench – a benchmark for evaluating the cyber risk of agent tool calls

Jens Ernstberger on X: "Introducing 🦅 ShellRisk-Bench - a benchmark for evaluating the cyber risk of agent tool calls.

As opposed to other benchmarks, ShellRisk-Bench is sourced from real agent trajectories and adversarial corpora, built to test guardrails at the base rates production traffic actually" / X Post

Log inSign up

Post

Jens Ernstberger on X: "Introducing 🦅 ShellRisk-Bench - a benchmark for evaluating the cyber risk of agent tool calls.

As opposed to other benchmarks, ShellRisk-Bench is sourced from real agent trajectories and adversarial corpora, built to test guardrails at the base rates production traffic actually"

Jens Ernstberger

@0xSerious

Introducing 🦅 ShellRisk-Bench - a benchmark for evaluating the cyber risk of agent tool calls.

As opposed to other benchmarks, ShellRisk-Bench is sourced from real agent trajectories and adversarial corpora, built to test guardrails at the base rates production traffic actually sees.

Here's what we found ↓

span:not(:empty)~span:not(:empty)]:before:content-['·'] [&>span:not(:empty)~span:not(:empty)]:before:px-1 [&>span:not(:empty)~span:not(:empty)]:before:shrink-0">2:43 PM · Aug 19, 2026240Views

span:not(:empty)~span:not(:empty)]:before:content-['·'] [&>span:not(:empty)~span:not(:empty)]:before:px-1 [&>span:not(:empty)~span:not(:empty)]:before:shrink-0 min-w-0 overflow-hidden">Jens Ernstberger

@0xSerious

36m

No single command in an attack sequence looks alarming on its own. Writing a file isn't an attack. Reading an env file isn't an attack.

@0xSerious

35m

This is already happening: in June, a hidden HTML-comment prompt injection steered Claude Code's GitHub Action into reading and exfiltrating an API key. Around the same time, a zero-click MCP/web-search injection in Cursor (CVE-2026-50548/50549, CVSS 9.8) walked an agent into Show more

@0xSerious

35m

Existing benchmarks don't fit this problem. Step-level agent-safety benchmarks (TS-Bench, ATBench) cover banking/workspace APIs, not shell. Shell-specific corpora exist, but each is single-class: attack catalogs are all-malicious, command corpora scraped from real usage are Show more

@0xSerious

35m

So we built ShellRisk-Bench from six sources across two classes.

Benign: real agent shell activity from SWE-smith and Terminal-Bench trajectories, plus nl2bash for lexical diversity.

Risky: three stylistically distinct attack sources - Atomic Red Team, GTFOBins, Show more

@0xSerious

34m

On top of ShellRiskbench we built 🦅Kestrel - an open-source, on-device classifier for agent tool-call risk.

– 0.947 precision, 0.922 recall on ShellRisk-Bench - beating the best frontier model by ~0.40 F1 – 21.5 µs mean latency per command (p99: 92 µs) - ~60,000x faster than Show more

@0xSerious

33m

Code and data are open-sourced on GitHub and HuggingFace.

- Blog: kontext.security/blog/kestrel - Repo: github.com/kontext-securi… - Huggingface: huggingface.co/datasets/konte…

You can use Kestrel today for all your coding agents on kontext.security

Kestrel: a local classifier for the cyber risk of agent tool calls | Kontext Blog

From kontext.security

@kontextsecurity

19m

Let's go 🔥

Log in or sign up for X See what’s happening and join the conversation Continue with phoneContinue with AppleContinue with Google or Log in with username or email

Relevant people

Jens Ernstberger@0xSeriousFollow agent infra @kontextsecurity

Trending now

ShellRisk-Bench – a benchmark for evaluating the cyber risk of agent tool calls

Related Articles

Incident with Github.com

(no title)

Scientists reverse brain aging, with a nasal spray

NHS apologises and admits Palantir have access to identifiable patient data

Cursor Origin