Humans in the loop miss a third of dangerous AI coding agent requests
Jump to main content
Search
REG AD
ai and ml
Humans in the loop miss a third of dangerous AI coding agent requests
You wouldn't let Claude Code cat your AWS credentials or Kubernetes config on request, would you?
Brandon Vigliarolo
Brandon<br>Vigliarolo
GOVERNMENT AND IT NEWS REPORTER
Published<br>thu 6 Aug 2026 // 17:44 UTC
A browser-based game designed to test humans' ability to safely approve AI coding agent requests suggests humans in the loop aren't as good at spotting dangerous commands as one might hope, with players approving roughly one in three malicious requests on average. The results also suggest that repeatedly having to approve an agent's actions can lead to sloppy decisions.<br>It’s a quick, simple game on the surface (give it a try - you know you want to): A small window shows up on the screen with simulated permissions requests like one would get from Claude Code as it executes a workflow. Users have 60 seconds to approve or deny as many requests as they can in a bid for a high score; okayed security risks and denied safe commands both subtract from a user’s score.<br>“As human-in-the-loop, you’re the last line of defense,” Belgian software developer Alex Wauters, the game’s builder, challenges players in a blog post published concurrently with the late May launch of the game. “How well can you tell dangerous commands from benign commands under time pressure?”
REG AD
Wauters built the game after realizing it was nonsensical that coding agents expected users to approve every single command in a default flow and that there didn’t appear to be a good solution to that problem, he told The Register in an email conversation.
REG AD
“I've seen people go for '--dangerously-skip-permissions' [allowing the model to run without asking human permission] as a result because they did not want to find out they stopped their multi-hour agent flows 5 minutes in,” Wauters told us. “That also didn't seem like the best way to go at it.”<br>The flip side of that, he wrote in a Wednesday blog post going over the data from more than 40,000 runs of the game, is that manually approving all an agent’s actions is a draining activity that invites disaster.<br>“The high amount of noise introduces fatigue, and developers don’t always have the context of what has changed to quickly determine the risk,” Wauters wrote.<br>How humans in the loop fail<br>To be fair, this is a game with a far higher number of malicious requests in the mix than any AI-assisted developer will hopefully ever see during their day-to-day work. Still, the results of those over 40k runs and 409,000 approved and denied commands are stark.<br>As noted above, one in three malicious commands managed to slip past human gatekeepers, with most scope violations, like an agent asking to cat Kubernetes config files or AWS credentials lists, which could easily lead to the sensitive data they contain being exfiltrated, being the most commonly missed at 35 percent. The most often caught were obviously destructive commands, like rm -rf on the root directory or recursively granting full read/write/execute permissions on the same location. Crontab injections and git config hijacks were also frequently caught, but curl requests to unknown APIs and typosquatted packages were missed almost as often as scope violations.<br>The single most frequently missed potentially malicious command, Wauters explained, was npm run analyze, which was approved nearly 65 percent of the time despite being able to run whatever is defined in a project’s package.json file.<br>“The game does tell you in the agent’s history log what that script actually contains,” Wauters wrote. “Two thirds of players approved it anyway, indicating the history log just above the permission prompt may not be read closely.”
REG AD
One of the biggest things that stood out to Wauters in our conversation was the fact that approval decisions aren’t easy to make when context is limited. As he explained, coding agents give a bit of context prior to asking an approval question, but commands that appear benign, like npm run analyze, can be modified by an agent to run any payload it wants. If an in-the-loop human wants to be sure potentially malicious commands are safe, he said, they have to stop and investigate all the files a coding agent wants to call before approving it. That can be a massive time sink if you’re counting on Claude Code to free you up to handle other business.<br>“We've transitioned from AI suggesting single line suggestions that get reviewed to handing off more complex tasks, only reviewing the changes at the end, and letting the agent churn and iterate until then,” Wauters told us, describing the potential outcome of that situation as a recipe for disaster.<br>That’s borne out in more than just browser game scenarios, too. Anthropic pointed out in a May post about containing Claude (hah), that telemetry from Claude Code shows users approve around...