Why Amazon hates 'human-in-the-loop' AI governance

Jump to main content

Search

REG AD

Security

Why Amazon hates 'human-in-the-loop' AI governance

VP Eric Brandwine explains people aren't all that great, actually

Jessica Lyons

Jessica<br>Lyons

Published<br>sat 20 Jun 2026 // 15:25 UTC

Humans tend to be “a little bit precious about humans,” according to Eric Brandwine, distinguished engineer and VP at Amazon Security.<br>We like to think we are all very good at our jobs, and we have high opinions of ourselves, he explained during a phone interview with The Register. “But when you actually get down to it, humans are not terribly consistent,” Brandwine said.<br>Humans, like AI agents and systems, are non-deterministic. Neither can be guaranteed to produce the same output given the same input twice. Both will make mistakes and even make stuff up. However, we’ve got millennia of experience dealing with humans and less than a decade with more modern LLMs and the AI systems built on top of them.

REG AD

“We know how humans fail,” Brandwine said. “We're comfortable with it. So human-in-the-loop isn’t necessarily the gold standard.”

REG AD

For years, vendors have told companies that the solution for dealing with any automated system was to put a human in the loop. That battle cry became much louder with the advent of modern AI systems and reached a fever pitch when enterprises started deploying agents into their IT environments.<br>More recently, however, big tech is changing the way it talks about agentic governance and rethinking the whole human-in-the-loop concept.<br>Normalization of deviance<br>In 2017, Brandwine gave a talk on the normalization of deviance at AWS’ annual re:Invent conference.<br>It’s a gradual process that happens when people in an organization take shortcuts, or don’t follow the established procedures or standards, and sometimes it occurs over years. As long as nothing catastrophic happens, this deviant behavior becomes the norm.

Eric Brandwine, distinguished engineer and VP at Amazon Security

“It’s a thing all humans fall prey to, and one of the most heartbreaking stories I read in this area was about emergency departments and emergency rooms,” Brandwine said during a phone interview with The Register. “You’ve got all these machines, and they’re all beeping. Your first day on the job, you jump every single time one of the alarms beeps – but the patient is fine. It's a spurious alarm. You go back to your station, you sit down, and over time, after enough of these false alarms, enough of these repeated beeps with no actual consequence, your discipline slips, and you stop responding. And eventually some tragic outcome occurs.”<br>This, he admits, is a very high-stakes example. And yet it’s a documented occurrence among healthcare workers, firefighters, and even Army pilots.

REG AD

“Literally, someone’s life is on the line, and people still struggle to maintain discipline,” Brandwine said. “That’s the human condition.”<br>Here’s how this all applies to agentic AI governance and security. Humans build LLMs and AI systems, and having a “human-in-the-loop” ensures that a person reviews the AI’s output and approves (or not) any actions before the AI performs them.<br>“If you put a human inside of this tight loop, and ask them to make approval decisions for agentic tools repeatedly, time after time, they'll do a good job,” Brandwine said. “And then they'll do an okay job. And pretty quickly they'll be doing a poor job.”<br>This is why at Amazon, “we’re not huge fans of human-in-the-loop,” he added. “It's something that you should use judiciously, where you absolutely need it. But it’s not something that you can do at high velocity. You will not get the results that you want to get.”<br>Big tech pulls the human-in-the-loop<br>Amazon isn’t the first or only tech giant to start talking differently about the role humans should play in agentic governance.<br>"It is very clear that we have moved from a human-led defense strategy, to a human-in-the-loop defense strategy, to an AI-led defense strategy that's overseen by humans," Google Cloud chief operating officer Francis deSouza told reporters during a press conference ahead of Google's annual Cloud Next shindig in April. "Our model for the future is an agentic fleet that does a lot of the routine cyber security work at a machine pace and then is overseen by humans."<br>Microsoft CEO Satya Nadella, in an X missive earlier this week, argued for “loop learning,” instead of having a human check an AI’s output at every step.

REG AD

“Companies need to turn their workflows, domain knowledge, and accumulated judgment into AI systems that improve with each use,” Nadella wrote. “Private evals should capture whether a model is actually improving against outcomes that matter to the business (not just external benchmarks!). Private reinforcement learning environments should let models grow stronger on real traces from inside the organization.”<br>Also this week, IBM execs called for human...