Why lawyers keep citing fake cases invented by AI | Scientific American
May 22, 2026<br>5 min read<br>Add Us On GoogleAdd SciAm<br>AI keeps inventing fake cases. Lawyers keep citing them
The trend of attorneys getting caught citing AI-hallucinated cases points to a broader problem: instead of checking AI’s work, people keep trusting it
By Steven Melendez edited by Eric Sullivan<br>Judges around the U.S. have sanctioned lawyers for filing court documents that include fake AI-generated citations.
John Pendygraft/AFP via Getty Images
In April the Alabama Supreme Court sanctioned an attorney who had filed legal briefs laden with inaccurate citations generated by AI, including numerous references to cases that did not exist. After being informed he had cited a made-up precedent in one filing, the lawyer promised it wouldn’t happen again—but then cited “nonexistent cases at the end of the very next sentence,” as a justice noted in a concurring opinion. At least one other lawyer was sanctioned that week for continuing to file AI-hallucinated material after being warned not to do so.<br>A database maintained by Damien Charlotin, a senior research fellow at the Paris School of Advanced Business Studies (HEC Paris), lists more than 1,400 cases where courts have addressed AI errors in the past three years, including filings by attorneys and self-represented litigants. As recently as last fall, Charlotin says, the list appeared to be growing exponentially. It’s since leveled off to a steady flow of exasperated judicial rulings. “For the past two or three months, we have reached a plateau of around 350, 400 decisions a quarter,” says Charlotin, who has also created an AI-powered reference checker called Pelaikan.<br>Courtroom proceedings are public, and lawyers face sanctions for false claims, making such errors comparatively easy to track. But uncaught errors in AI-generated material have also ensnared journalists, software developers, academic researchers and government consultants, some of whom have been well aware of AI’s fallibility. On May 19 the New York Times reported that the author of The Future of Truth, a book about how AI is shaping discourse, acknowledged his text contained more than a half-dozen fabricated or misattributed quotes produced by the technology.<br>On supporting science journalism<br>If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.<br>The pattern emerging across these cases is that people keep trusting AI’s answers even when they know the systems can be wrong. So far, that misplaced trust has led to dismissed legal appeals, attorney fines, fired journalists and software outages. Experts warn the stakes will rise as AI becomes more deeply embedded in professional work.<br>“Humans essentially have a tendency to believe that machines have more knowledge than they do, don’t break and are infallible,” says Alan Wagner, an associate professor of aerospace engineering at Pennsylvania State University.<br>AI also appears to inspire a particular kind of trust. It can generate answers that are realistic-sounding but false in a way humans seldom do—and people, it turns out, can find its guidance unusually believable. A study published this past February asked participants to complete an image classification task with guidance they were told came from either humans or AI. The guidance—no matter where it came from—was right only half the time, but among participants who were told the advice came from AI, those with positive attitudes toward the technology performed worse than those who held less favorable views. No such effect appeared when participants were told the advice came from humans.<br>“The results suggested that AI guidance has a quite specific ability to engender biases,” says study co-author Sophie Nightingale, a senior lecturer in psychology at Lancaster University in England.<br>Research co-authored by Wagner suggests the problem could extend well beyond office work into life-or-death scenarios. In experiments inspired by drone warfare, his team asked participants to categorize images as civilians or enemy combatants and to choose whether to fire a missile at each potential target. A robot then provided feedback on each classification—feedback that was, in fact, random—and though participants’ initial assessments were mostly accurate, they reversed their views in most cases where the bot disagreed. The scenario was a simulation, but participants were “shown imagery of innocent civilians (including children), a UAV [uncrewed aerial vehicle] firing a missile, and devastation wreaked by a drone strike,” according to the paper. They seemed to take the task seriously, says study co-author Colin Holbrook.<br>“I think that’s the...