How well does AI peer review work? - by Paul Litvak

In One Lifetime

SubscribeSign in

Meta Science<br>How well does AI peer review work?<br>A new evaluation benchmark

Paul Litvak<br>Aug 10, 2026

15

Share

Summary

Claude and I planted 100 known errors into 10 open-access psychology papers and then ran them through frontier models and two commercial AI review tools. In brief:<br>The best single system caught 71 of 100 errors, while the worst caught 30.

Pooling every system’s output caught 93 of 100. Models are only partly correlated in the errors they find, making ensembling a big lever for finding issues in papers. Check your papers against multiple models!

Seven errors could not be caught by any system. All were omissions — information deleted from a paper rather than mistakes inserted into it.

Refine.ink contributes more unique catches than any other single system, though it’s expensive.

I didn’t measure false positives and I don’t know how this error distribution compares to the distribution of errors in real papers.

I’ve made the papers, errors, model outputs, and the full experiment log public. I hope people can build on this work to create a comprehensive eval benchmark across disciplines.

Introduction

There has been a proliferation of AI tools that purport to do peer review. This is coming at the exact moment when the number of papers is also starting to grow due to AI assistance. All this new knowledge is about to be created, but humans lack the collective attentional capacity to review it all! We are on the precipice of a revolution in collective knowledge infrastructure, where the next step is going to surpass the span of the human mind itself. But we aren’t yet able to fully turn ourselves over to this new technological infrastructure — after all, how can we trust it? How can we know that what it produces is right, across a given field of knowledge, as the frontier of AI continues to move (and is extremely jagged)? This problem is most urgent in the sciences. I think most scientists following these tools closely would agree that AI peer review, the ability of AI to evaluate a given piece of scientific research, is beginning to get quite good1. But how good? And how do we measure when it’s time to hand over the reins?<br>I was recently talking about this topic with a friend who is a medical school professor — and he made the point that the new science isn’t going to be legible. The best human minds in medicine have an increasingly narrow scope of in-depth knowledge of a specific subject area. Science will soon outpace the ability of any individual human mind, and the time it has available, to comprehend it. We will likely need a shared, continuously updated record of what is actually known in a field — a living context layer for science — if we as humans have any hope of keeping up. As an example, the concept of diabetes we have right now is merely a shorthand2 for a much more complex phenomenon — it might be better represented by a 90-dimensional vector spanning dozens of biomarkers instead of a single threshold like A1C. Correspondingly, the standard of care would stop being a threshold rule a clinician can hold in their head and become a function computed over that whole representation. To this point, a lot of disagreement on science funding has hinged on the degree to which we allow science to be legible to democratic review or not.<br>Creating a benchmark

To that end, I was brainstorming with Claude on developing good evaluations of peer review software. Most systems work by ensembling, having the LLMs judge each other’s output. That, as we’ll see, is a good strategy, since different LLMs are surprisingly uncorrelated as far as the errors they can catch. Nonetheless, this isn’t satisfying since we don’t know whether even pooled models miss some categories of errors. Using human peer review as the gold standard, as any professor will tell you, is also unsatisfying, since human peer review varies widely in quality. Of Claude’s suggestions, one stood out — inserting errors into papers and seeing whether a given AI system can spot them. Although this doesn’t measure false positives (identifying errors that aren’t really errors), it at least provides a grounded baseline for measuring true positives and false negatives.<br>Convinced by the viability of this approach, Claude and I set out3 to modify 10 existing open-access psychology papers4, inserting errors into them both by deleting key clauses and altering others. We created a taxonomy of errors in experimental psychology papers, drawn from existing taxonomies like the INSPECT-SR checklist. We ended up with a 62-category taxonomy of types of errors, ranging from statistical errors, construct validity issues, causal inference, generalizability, etc. Then we created 100 errors, 10 per paper, and inserted them. The first iteration was a bust — the errors were too basic and the benchmark was immediately saturated when we ran the papers through foundation...