ErrataBench - Which LLM is the best proofreader?

ErrataBench<br>v1GitHub<br>Which LLM is the best proofreader?<br>Benchmarked 100 model variants across 3196 runs with --samples 3 --chunk-size 2000 --max-turns-per-chunk 3Run statsTotal runtime9d 3h 34mTotal cost$1,684<br>Updated Jul 24, 2026, 12:41 PM - commit 2ea3243 - art@revise.io

Proofreading with LLMs

LLMs make fast, cost-effective proofreaders - but which models are the best? How thoroughly can they find and fix errors, and how efficiently?<br>ErrataBench measures all of this by running a simple agent loop over large samples of text, each containing a wide variety of writing errors. We ask models to find and fix as many problems as they can find using "find and replace" tools.

Overall Rankings

The chart below is an overall ranking of success rate. What percentage of errors did the model find and correctly fix?<br>For issues not fixed successfully, we make a distinction between omissions ("Not Addressed") and attempted fixes which were rejected as incorrect by the judge ("Bad Fix").<br>Models were all given the same prompt, tools, text chunks, and number of turns.

Error Fixed<br>Bad Fix<br>Not Addressed

OpenAIAnthropicGooglexAIQwenDeepSeekMiniMaxMoonshot AIXiaomiZ-ai

Best Variant Only

Show 25 more

Efficiency

Here we compare the overall success rate with other dimensions like speed and cost. Only the best reasoning variant of each model is shown by default; toggle off "Best Variant Only" to see all data points.<br>This two-dimensional comparison gives the full picture between performance and cost, depending on what is important to you. Tool Call Efficiency measures how surgical the model was with its changes.<br>Use "Highlight Dominant Models" to see which models dominate on the axis you're viewing.

Cost Efficiency<br>Errors fixed per $1 USD of spend<br>(Higher is better)<br>Speed<br>Errors fixed per minute of runtime<br>(Higher is better)<br>Turns Used<br>Number of turns used to finish<br>(Lower is better)<br>Tool Call Efficiency<br>Errors fixed per 100 tool-output characters<br>(Higher is better)<br>Consistency<br>Median of per-dataset score ranges<br>(Lower is better)

OpenAIAnthropicGooglexAIQwenDeepSeekMiniMaxMoonshot AIXiaomiZ-ai

Only 90%+Best Variant OnlyHighlight Dominant ModelsShow Outliers

Methodology

Dataset<br>ErrataBench tests models against a dataset of English text from various sources (literature, law, technical manuals). Each source has been altered with a wide variety of writing errors, across several categories. These errors are tracked in a corresponding file.<br>Source text is altered using a corruptor model + review model pair; the corruptor is instructed to make small changes to the text to insert errors of specific types, sampled from the error category taxonomy. The review model reviews the changes as a second opinion before they are accepted into the dataset.<br>Agent Loop<br>The benchmark runs a simple agent loop over each altered text, showing the model chunks of up to 2000 words at a time while keeping whole paragraphs together. The model is simply asked to proofread the text carefully, fixing errors while preserving meaning and details. The model is not given any specifics about what types of errors are hiding in the text or how many of them there are.<br>The agent loop gives the model up to 3 turns with each chunk, during which the model can modify that text using simple tools: find_and_replace (to make surgical changes) and replace_paragraph (for wider rewrites). If the model changes the text to what it was originally, the error is immediately counted as fixed. If the model changes it to something else, an LLM judge is used to decide if that alternative is also correct.<br>Metrics<br>The headline Fix rate score is an equal-weighted per-dataset mean, not a pooled average over all runs. For each source dataset d, ErrataBench averages the successful-run quality scores for that dataset to get mean(q_d), then averages those dataset-level means across datasets. That keeps datasets with more successful repeats from counting more heavily than datasets with fewer.<br>The other scatterplot metrics are derived similarly. Cost Efficiency is corrected issues per USD at the run level, then averaged by dataset and across datasets.Speed is corrected issues per minute of run time, aggregated the same way. Tool Call Efficiency is shown as issues fixed per 100 benchmark-scoped tool characters, which is an inverted display of the underlying tool-chars-per- resolved-issue metric. Turns Used is the mean average turns per chunk across datasets.<br>The scatterplot's consistency metric is median_d range(q_d), the median per-dataset range of quality. For each dataset, ErrataBench measures the spread between the highest and lowest successful-run quality score, then takes the median of those dataset-level ranges. In the UI this appears as Consistency. Lower values mean the model's proofreading quality is more consistent from run to run, while the median makes the metric less sensitive to a few noisy source texts.<br>Reproducing...