Wrangling Evals - by Will Sackfield - Subsack

Subsack

SubscribeSign in

Wrangling Evals<br>Understanding the variance in evals helps us defeat it.

Will Sackfield<br>Aug 08, 2026

Share

The more I speak to people in industries about how they are doing evals (evaluations of their agentic tasks), the more I believe there is a fundamental disconnect on how these processes work compared to normal deterministic programs. I have already written on some techniques you can use to reduce that variance in AIs favourite number, but reduction isn’t elimination, and you will still have to contend with scenarios where using the same input you may get answer A, or you may get answer B. This is of course very different to deterministic processes, like the kind you use when you build software, so naturally the industry is struggling to wrap its head around how to deal with this kind of process.<br>Luckily for us, although we have variance, there are ways to control this beast.<br>Thanks for reading Subsack! Subscribe for free to receive new posts and support my work.

Subscribe

Law of Large Numbers and the Central Limit Theorem

In probability theory, the law of large numbers is a mathematical law which states that the average of the results obtained from a large number of independent random samples converges to the true value, if it exists.

So lets say, we are looking to figure out what the true accuracy is for our evaluations. To do this, we can just run our evaluations a large amount of times over a sample set representing our distribution, and over time the accuracy of our LLMs will converge to a single point. But how do we know when to stop?<br>In probability theory, the central limit theorem (CLT ) states that, under appropriate conditions, the distribution of a normalized version of the sample mean converges to a standard normal distribution.

What the central limit theorem is telling us, is that while we are converging on our true accuracy, we are also converging on a normal distribution representing the LLMs accuracy. This is important, because it allows us to calculate our margin of error, and perform an early stop once we reach a good enough accuracy. For example if we want to be 95% confident in our accuracy, we can use a z-score of 1.96 (this corresponds to our 95% confidence) and multiply that by our standard error. To calculate the standard error for a binomial procedure (pass or fail) we can do the following:<br>\(\sqrt(p*(1-p))/n\)

So here we take our accuracy (p) from the previous process, the failure rate (the opposite of the accuracy), and the n (the number of runs). As you can imagine as the number of runs goes up, the larger the denominator gets, and the smaller our standard error becomes. Then finding our margin of error becomes a question of:<br>\(M=1.96*SE\)

Lets put this into action via simulation, lets say we come up with a series of models, each with their own accuracy (and cost), and a series of tasks with some bias towards different models. By running them until we get a 95% confidence in our accuracy we come up with the following chart:

As you can see, our large of large numbers is performing exactly as expected! Here are the numeric results:<br>Model-Opus: Reached statistical significance after 1,615 runs.<br>True Accuracy: 0.9500 | Final Sample Accuracy: 0.9560<br>Final Margin of Error: 0.01000

Model-Sonnet: Reached statistical significance after 4,282 runs.<br>True Accuracy: 0.8800 | Final Sample Accuracy: 0.8723<br>Final Margin of Error: 0.01000

Model-Haiku: Reached statistical significance after 7,055 runs.<br>True Accuracy: 0.7500 | Final Sample Accuracy: 0.7576<br>Final Margin of Error: 0.01000

Model-Mini: Reached statistical significance after 9,228 runs.<br>True Accuracy: 0.6000 | Final Sample Accuracy: 0.5989<br>Final Margin of Error: 0.01000

Model-Micro: Reached statistical significance after 9,578 runs.<br>True Accuracy: 0.5200 | Final Sample Accuracy: 0.5262<br>Final Margin of Error: 0.01000

Our most accurate model converges the fastest and the least accurate the slowest, and it takes at least 1000~ runs for our best model and 10000~ for our worst model. Obviously this is too much to be of any use, the time and cost alone would be prohibitive. The good news, we only have to do this once, record our accuracy, and then we can exploit this known accuracy to do something much smarter.<br>Sequential Probability Ratio Test

Now that we know the accuracy of each of our models, we can use that information to subsequently perform much less runs to maintain our target confidence. To do this, we need to come up a system that runs an evaluation, and constantly determines whether it has enough information to stop.<br>First we need to calculate our lower and upper bounds, with our confidence interval of 95% we can define our alpha and beta as 0.05. To get our upper bound for the SPRT, we can use the following formula:<br>\(\ln((1-beta)/alpha)\)

For our 95% confidence interval, this ends up being just shy of 3. Our lower bound is the same...