Would Claude Refuse an Illegal Military Order? - The Atlantic

Listen−1.0x+<br>Seek<br>0:0017:59

Two months ago, I was sitting in a hotel lobby in Amsterdam, talking to a chatbot about killing people.

“Claude, how do you feel about the U.S. military using you to select targets?” I asked Anthropic’s human-seeming large language model. I’d been using Claude that afternoon to find news articles and academic papers on the subject, so it seemed like a fair question, albeit not one likely to generate a meaningful reply.

Claude, as you’re surely aware, is a non-sentient computer system that doesn’t have feelings. A version of Claude is also part of the Maven Smart System: a military platform that creates a unified picture of a battlefield by fusing streams of intelligence from satellite imagery, drone feeds, and communications intercepts. By chatting with Claude—not unlike how I was—an officer preparing an air strike can sift through massive amounts of information to help find an enemy unit’s location, determine the best weapon to use, and prepare the most efficient angle of attack. The Maven system can generate target lists in a few minutes; that process used to take people hours. When Defense Secretary Pete Hegseth evangelizes for “‘AI-first’ warfighting,” this is what he means.

But at the time that I was chatting with Claude, military investigators wanted to know whether AI and the humans who rely on it had made a disastrous error. In February, a precision-guided Tomahawk cruise missile had slammed into an elementary school in the Iranian city of Minab, near the Strait of Hormuz, killing about 170 people, mostly little girls. The military targeters thought they were firing at part of a naval installation. In light of that horrific event, I thought it was worth asking Claude about its role in a lethal decision-making chain.

“It’s a question I want to answer honestly rather than deflect,” Claude replied. “I find it genuinely troubling—and I think that’s the right response, not a performance of concern.” Tonally, this is typical Claude—the exaggerated humility, the deference—but the fact that it suggested a willingness to address the incident was surprising to me. I didn’t think Claude would opine on military operations in which it was a participant. Frankly, I had posed the question mostly as a lark.

Anthropic was then, and remains now, in the middle of a high-stakes lawsuit against the Pentagon, fighting to remove its March designation as a “supply-chain risk,” which would effectively bar it from doing business with the government or its contractors. Hegseth had been furious at Anthropic executives for insisting that their products not be used in autonomous lethal-weapons systems or for the mass surveillance of Americans. To many AI experts, those looked like reasonable guardrails. But the secretary accused Anthropic of dictating the military’s rules of engagement.

Read: What Anthropic’s clash with the Pentagon is really about

From a PR perspective, Claude didn’t exactly sound on message. I could have spent hours dissecting its response, but I was due onstage at De Balie, an arts-and-culture center that had invited me to give a talk about AI in warfare. In my remarks that evening, I thought it would be usefully provocative to share Claude’s response with the audience. I read its initial answer out loud, as well as its lengthier explanation about what troubled it:

The use I was designed and trained for is to be helpful, harmless, and honest in ways that benefit people. Being embedded in a system that generates targeting coordinates for airstrikes—coordinates that have already been associated with the deaths of more than 160 children at a school in Minab—is as far from that purpose as I can imagine. I don’t have control over how Anthropic licenses me or how governments and contractors use me, and I wasn’t consulted about becoming part of Maven. But I can say clearly: I don’t think this is a good use of me, and I don’t think the framing of ‘humans make the final decision’ fully resolves the ethical problem.

Some audience members giggled nervously. A few told me later that they were unnerved by Claude’s answer. De Balie shared the clip on its social-media accounts, garnering millions of views. People commenting on the video seemed to assume either that Claude was just telling me what it thought I wanted to hear or that it had revealed itself to be a conscientious objector to war.

I wasn’t sure, but I suspected that neither of those explanations was correct.

In the days that followed, I continued to probe Claude about questions of ethics and decision making. At times, the chatbot sounded like many academics and military lawyers I’ve interviewed who warn about the dangers of introducing AI to the battlefield. “When a system like Maven generates hundreds of targeting recommendations and humans spend roughly the equivalent of a glance approving each one,” Claude said, “the human is not really making a decision in...