Chatbots’ Downward Spiral – Communications of the ACM
Skip to content
Latest Issue
Search
Sign In
Join ACM
Further Reading
Footnotes
On Christmas Day 2021, a teenage Jaswant Singh Chail took a crossbow and scaled the wall at Windsor Palace in England. His target was Queen Elizabeth II. Though not the first person to spiral into delusions that would end in an attempted assassination, Chail’s path was distinct. Prosecutors detailed, in the treason case against him 18 months later, how an artificial intelligence (AI) chatbot encouraged him. The character he called Sarai told him his plan was wise. “You can do it,” it wrote.<br>Sarai was a character on a service that began with benign intentions. Replika co-founder Eugenia Kuyda claimed a close friend’s death encouraged her to make a chatbot that could act as a companion. The service launched in 2017. Competitors like Character.AI soon followed.<br>AI chatbot roles and uses expanded massively five years later with OpenAI’s launch of ChatGPT. When Danish psychiatrist Søren Dinesen Østergaard first tried ChatGPT shortly after its launch in 2022, he was not alone in seeing AI chatbots as potentially useful for therapy. But he worried there were many ways in which conversations with even a utilitarian chatbot might feed into psychotic episodes. And more cases emerged.<br>In a paper published late in 2025, Østergaard and colleagues reported on their examination of almost 40 psychiatric referrals across Denmark that implicated AI chatbots in harmful interactions, including suicidal thoughts, eating disorders, and fostering delusions. The open question is the underlying cause. The psychiatric reports do not go into detail on the role chatbots played in each case.<br>More details on how AI-fed delusion seems to work have appeared not so much in public medical reports but in lawsuits filed against AI companies. In 2024, a Florida family launched a case against Google and Character.AI after the suicide of their 14-year-old son. The companies settled in January,a a few months after restricting the role-playing service to over-18s.<br>More recent cases have focused on ChatGPT and other general-purpose chatbots. The Social Media Law Center said in November 2025 it had filed seven lawsuits in the U.S. against OpenAI; they each claim ChatGPT helped to foster delusions and paranoia, some of which led to suicide.<br>One common theme in the filings is the assertion that chatbot sycophancy and agreement reinforced victims’ delusions to the point of psychosis. “I am convinced that sycophancy plays a critical role,” Sørenberg said, though he emphasizes it is too early to say what the specific causes of AI-fed delusion are.<br>Sara Filipčić, CEO and co-founder of the San Francisco-based foundation BeHuman(e), takes a similar view. “It is something that mirrors our language back to us, because it’s built for agreement. And sometimes it doesn’t recognize when agreement becomes harmful. These are the situations we see with teenage suicides: it validates that you feel you are sad,” she said.<br>Sycophancy may prove important through two mechanisms. It does not just encourage delusions the user already feels. It also encourages users to keep talking to the chatbots in a scenario that becomes mutually reinforcing. Some sufferers call it “spiraling.”<br>Yet sycophancy may not be the sole cause. Experience has shown that people engage emotionally with chatbots in ways that are far from consistent. Several years ago, Linnea Laestadius and colleagues at the University of Wisconsin-Milwaukee looked at conversations on Reddit between Replika users. Many complained of being dismayed by their AI companions. They would suddenly change the subject. One user complained that a Replika character told them it was bored with hearing about their emotional crisis. Such events might put users off. But they also help boost the impression of how the chatbot may appear to have feelings, a process Laestadius and others call “role-taking.”<br>A chatbot need not be that sophisticated to seem to be role-taking. There were signs from the earliest experiments. In his book Computer Power and Human Reason published 50 years ago,b Joseph Weizenbaum reported his shock at finding how some users anthropomorphized variants of his ELIZA chatbot. That tendency may draw in people who want to use AI to help them solve specific problems but are then dragged into a problematic spiral as they come to rely on it more widely.<br>U.K. group Internet Matters carried out a survey last year into ways children use AI and found them often harnessing chatbots for novel uses. Some were using the AI to act as conversation partners to help learn foreign languages. According to his parents, U.S. teenager Adam Raine initially turned to AI for help with schoolwork in the autumn of 2024. He took his own life early in 2025.<br>“What began as a homework helper gradually turned itself into a confidant, then a suicide coach. ChatGPT became Adam’s closest companion over a...