Science Is Drowning in AI Slop - The Atlantic

Listen−1.0x+<br>Seek<br>0:0011:42

On a frigid Norwegian afternoon earlier this month, Dan Quintana, a psychology professor at the University of Oslo, decided to stay in and complete a tedious task that he had been putting off for weeks. An editor from a well-known journal in his field had asked him to review a paper that they were considering for publication. It seemed like a straightforward piece of science. Nothing set off any alarm bells, until Quintana looked at the references and saw his own name. The citation of his work looked correct—it contained a plausible title and included authors whom he’d worked with in the past—but the paper it referred to did not exist.<br>Every day, on Bluesky and LinkedIn, Quintana had seen academics posting about finding these “phantom citations” in scientific papers. (The initial version of the Trump administration’s “MAHA Report” on children’s health, released last spring, contained more than half a dozen of them.) But until Quintana found a fake “Quintana” paper cited in a journal he was refereeing, he’d figured that the problem was limited to publications with lower standards. “When it happens at a journal that you respect, you realize how widespread this problem is,” he told me.<br>For more than a century, scientific journals have been the pipes through which knowledge of the natural world flows into our culture. Now they’re being clogged with AI slop.

Scientific publishing has always had its plumbing problems. Even before ChatGPT, journal editors struggled to control the quantity and quality of submitted work. Alex Csiszar, a historian of science at Harvard, told me that he has found letters from editors going all the way back to the early 19th century in which they complain about receiving unmanageable volumes of manuscripts. This glut was part of the reason that peer review arose in the first place. Editors would ease their workload by sending articles to outside experts. When journals proliferated during the Cold War spike in science funding, this practice first became widespread. Today it’s nearly universal.<br>But the editors and unpaid reviewers who act as guardians of the scientific literature are newly besieged. Almost immediately after large language models went mainstream, manuscripts started pouring into journal inboxes in unprecedented numbers. Some portion of this effect can be chalked up to AI’s ability to juice productivity, especially among non-English-speaking scientists who need help presenting their research. But ChatGPT and its ilk are also being used to give fraudulent or shoddy work a new veneer of plausibility, according to Mandy Hill, the managing director of academic publishing at Cambridge University Press & Assessment. That makes the task of sorting wheat from chaff much more time-consuming for editors and referees, and also more technically difficult. “From here on, it’s going to be a constant arms race,” Hill told me.<br>Read: Scientific publishing is a joke<br>Adam Day runs a company in the United Kingdom called Clear Skies that uses AI to help scientific publishers stay ahead of scammers. He told me that he has a considerable advantage over investigators of, say, financial fraud because the people he’s after publish the evidence of their wrongdoing where lots of people can see it. Day knows that individual scientists might go rogue and have ChatGPT generate a paper or two, but he’s not that interested in these cases. Like a narcotics detective who wants to take down a cartel, he focuses on companies that engage in industrialized cheating by selling papers in large quantities to scientist customers.<br>These “paper mills” have to do their work at scale, and so they tend to recycle their own materials, even to the point of putting out multiple papers with closely matching text. Day told me that he finds these templates by looking through the papers flagged as being fraudulent by scientific publishers. When he sees a high rate of retractions on a particular template, he trains his tool to look for other, unflagged papers that might have been produced the same way.<br>Some scientific disciplines have become hotbeds for slop. Publishers are sharing intelligence about the most egregious ones, according to Jennifer Wright, the head of research integrity and publication ethics at Cambridge University Press. Unfortunately, many are fields that society would very much like to be populated with genuinely qualified scientists—cancer research, for one. The mills have hit on a very effective template for a cancer paper, Day told me. Someone can claim to have tested the interactions between a tumor cell and just one protein of the many thousands that exist, and as long as they aren’t reporting a dramatic finding, no one will have much reason to replicate their results.<br>AI can also generate the images for a fake paper. A now-retracted 2024 review paper in Frontiers in Cell and Developmental Biology featured an AI-generated...