The uncritical adoption of AI in science is alarming — we urgently need guard rails
Skip to main content
Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain<br>the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in<br>Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles<br>and JavaScript.
Advertisement
The uncritical adoption of AI in science is alarming — we urgently need guard rails
Download PDF
Bluesky
Computerized automation can hamper the acquisition of skills and expertise. Credit: Westend61/Getty
You have full access to this article via your institution.
Download PDF
The scientific community is adopting artificial-intelligence tools, especially large language models (LLMs), at an astonishing speed. LLM-assisted paper writing has drastically increased over the past three years1 and researchers have sought to incorporate semi-autonomous agents into their workflows. However, the rapid and uncritical adoption of AI in science comes with significant risks2. Several problems are already apparent: papers that use AI tools focus on a narrower set of established research questions3, and in some cases have been evaluated to have less scientific merit4, than do studies that do not rely on AI.<br>Why an overreliance on AI-driven modelling is bad for science
Moreover, as AI automates routine scientific tasks, concerns about the erosion of training opportunities for early‑career researchers remain largely unresolved and often unacknowledged. Conventionally, scientific training has combined formal instruction in facts and methods with the gradual acquisition of tacit knowledge through hands‑on, entry‑level work. Scholars in science and technology studies have repeatedly demonstrated that scientific texts alone do not fully communicate knowledge; instead, crucial know‑how is embedded in research communities and transmitted through apprenticeship and practice5.<br>This tacit knowledge — for example, of what constitutes ‘reasonable’ data, or the details of a technique that are difficult to articulate in a methods section, or whether a result is consistent with the existing literature — is essential if a researcher is to supervise AI‑assisted workflows effectively in the future. If AI systems increasingly replace entry‑level scientific labour, trainees might never develop these skills, potentially leaving the next generation of scientists ill prepared to oversee AI‑driven research responsibly.<br>These trends demand that scientists reckon with the purpose of scientific institutions. Is our goal simply to build a collection of scientific facts, or to also cultivate a living, evolving community of scientific knowers? If AI tools accelerate the former goal while threatening the latter, how should scientists proceed? Drawing on emerging evidence about AI’s effects on scientific practice, here we highlight key risks and outline potential remedies.<br>More output, less understanding<br>The AI industry has aggressively marketed LLM products to scientists as technologies for increasing productivity. Some researchers have embraced this promise, touting these products’ ability to ‘supercharge’ their academic writing6.<br>Is AI leading to a reproducibility crisis in science?
If we take productivity to mean the output of scientific papers, AI products have undoubtedly delivered on the promise1,7, with far-reaching effects. Online preprint repositories such as SocArXiv and PsyArXiv imposed temporary moratoriums on AI papers or updated their reviewing policies; arXiv changed its policy and no longer accepts computer-science position papers (which are relatively easy for LLMs to churn out), and the US National Institutes of Health began limiting grant proposals to six per year per principal investigator (PI).<br>Alongside increased productivity, there is evidence that scientists who use AI tools in their work receive more citations and advance more quickly to PI roles than do those who do not3, with people who have English as their first language benefiting disproportionately8. It’s noteworthy, however, that the career benefits of AI products are most apparent in quantitative metrics, such as publication and citation counts. Such measures cannot account for whether these benefits accrue because of scientific contribution or as a result of connections with a hyped field. It would therefore be a mistake to take these metrics as representing a win for science, without looking more closely at the quality of this increased productivity.<br>A robot learns to emulate human behaviour at a facility in Hefei, China.Credit: Zhang Dagang/VCG via Getty<br>More is not necessarily better, and there is growing evidence that LLM-driven increases in scientific productivity often come at a cost. Most obvious is the introduction of ‘AI slop’ into the literature, including...