Study Claiming AI Helps Students Learn Retracted - Plagiarism Today

In May 2025, Jin Wang & Wenxiang Fan published a study in the Springer Nature journal Humanities & Social Sciences Communications entitled "The effect of ChatGPT on students’ learning performance, learning perception, and higher-order thinking: insights from a meta-analysis."

The meta-analysis made a very bold claim: that students who use ChatGPT (or other AI tools) perform better than students who don’t. In short, that despite the concerns that educators have about AI, that students learn faster and better when they use AI.

The study went on to become incredibly popular. Though it was cited over 500 times by other researchers, the study was especially popular on social media. By the end of its run, it was in the 99th percentile in terms of its overall attention score.

That is largely because the study became very popular in the ed tech space as technology companies pushed for more ways to integrate AI into education.

However, the study has now been retracted. It was originally retracted by the journal, in May of this year. But we now have more insight into why the study was retracted and that likely means for those who made decisions based (at least in part) on the study.

To that end, it honestly doesn’t look good for the state of research in this space.

Background on the Study

The study was conducted by Jin Wang & Wenxiang Fan, both from Hangzhou Normal University in Hangzhou, China. It was published in the journal Humanities & Social Sciences Communications, a Springer Nature journal, in May 2025.

The study is a meta-analysis, meaning that it is examining other studies in a bid to find that there is a trend in the research that had been performed up to that date. To that end, the researchers examined 51 studies that were published between November 2022 and February 2025.

The results of the analysis claimed that there is a "large positive effect" (g = 0.867) on learning performance when students use ChatGPT. There were also "moderate positive effects" on learning perception and higher-order thinking (g = 0.456/0.457).

The study further claimed that impacts required a duration of 4-8 weeks and worked best with problem-based learning and when ChatGPT was used as a "flexibly integrated into teaching as an intelligent tutor, learning partner, and educational tool."

This outcome was immediately seized upon by the ed tech space, who shared the study widely on social media and used it to promote AI products for the classroom.

However, also almost immediately, the study came under fire. Less than a month after it was published, Ben Williamson published a blog post in which he claimed that the study (along with a similar one) was "methodologically flawed, overhyped and misleading."

The very next month, Magnus Ingebrigtsen and Marko Lukic, both from The Arctic University of Norway, published an article entitled "Substantial Errors Invalidate The Conclusion From Wang And Fan (2025) About ChatGPT And Learning Outcomes." (Preprint available here)

The duo highlighted a variety of issues with the study. This included dubious studies that were included in the analysis as well as problems with the analysis itself.

In the end, they requested that the study be retracted and it was, with Springer Nature confirming that it was retracted largely due to their efforts.

So why was the study retracted? Simply put, because its conclusions were not supported by the data.

The Cause of the Retraction

The retraction notice states that the study was retracted due to "discrepancies in the meta-analysis" that "undermine the Editor’s confidence in the validity of the analysis and the conclusions drawn from it."

Those issues, according to the paper from Ingebrigtsen & Lukic, fall mostly into two categories. Issues with the analysis and issues with the studies that were included in the analysis.

According to the authors, the study was fraught with issues including discrepancies over the number of included studies, failure to weigh the studies appropriately and overreporting the found effects.

One of the most serious claims on that front was the aforementioned g=0.867 effect size for learning performance. According to Ingebrigtsen & Lukic, the study’s own forest plot shows that the effect size should have been closer to 0.5, even with all the other issues.

On the other side of the coin, the authors found that the original meta-analysis included a retracted study, several studies that were mislabeled, studies that were too small or had reported effect that were too large to be seen as reliable.

To make matters worse, many of the studies included in the analysis did not properly control for confounding variables, making them even less reliable.

In short, Ingebrigtsen & Lukic found that the study could not support its conclusions. As such, they recommended that the study be retracted. According to Springer Nature, the original authors never responded...