Pitfalls of Benchmarking on Modern Systems

Pitfalls of Benchmarking on Modern Systems · Stefan-Marr.de

Stefan Marr

Full Professor (Universitätsprofessor)

I head the Institute for System Software at the Johannes Kepler University Linz. My work focuses on programming language implementation techniques, compilation, complex concurrent systems, and tooling.

I am personally interested in making interpreters faster, detecting and preventing concurrency bugs at run time, and use modern language runtime systems to provide programmers with better tooling.

For contact details, please see my staff page.

Modern computer systems are quite fascinating. They are highly complex, run a lot of software, and their performance characteristics are hard or impossible to predict.

Earlier this year, I gave a lecture on benchmarking, which for lack of generic and always-applicable advice, I started with a brainstorming session about its pitfalls. Since systems have become so complex that we rarely know what happens exactly. We are prone to simply guess what’s going on. Or rather, we should hypothesize and verify our hypothesis. Still, there might be many more causes for what we see than we can initially think of.

To illustrate this, I’ll use some fictitious benchmark results, which are fairly close to what one would see in reality, and there are usually many possible explanations for the observed behavior. This often means, only once we know what caused a specific artifact, we can make progress with understanding what we set out to understand in the first place.

The Scenario: A “Reasonably Deterministic” Workload

Let’s start with our hypothetical/fictitious benchmark. One rather strong assumption we are going to make is that our benchmark is “reasonably deterministic”. Thus, when we run it multiple times, it pretty much does the same thing. To make this more concrete, let’s assume our benchmark does what one of the very first business applications did: payroll. We have a program that generates PDFs with the monthly payslips. We run our benchmark on a Linux from 2026 and since our hypothetical benchmark is written in Java, we use the HotSpot JVM and JDK 26.

A First Run

We let this benchmark run for 8 iterations within the same JVM process and visualize all our data points with a scatter plot. On the y-axis we show run time, which means lower is better. On the x-axis, we have the number of the specific iteration.

Figure 1: A first run of our benchmark. The x-axis shows the iterations, and the y-axis the run time. Thus, lower is better. We see that each subsequent iteration got faster until iteration 4, and then we stayed at the same performance level.

On the plot we can see that iterations 2, 3, 4 are each a bit faster than the previous one, and then we stay at the same level of performance.

Given that we use a language implementation with just-in-time (JIT) compilation, this is what we would roughly expect. The JIT compiler manages to optimize the code we are executing step by step, and we see that it improves performance.

A Second Run, Same Benchmark, Nothing Changed

We ran the exact same setup a second time, getting 8 new data points. But as we can see in Figure 2, the benchmark is faster in iteration 3 and following! What happened here?

Figure 2: A second run of the same benchmark, with the same setup. Surprisingly, it is quite a bit faster than on the first run!

This is where we now start hypothesizing.

What could it be? It could be that the operating system decided to give it different physical memory, or run it on a different kind of core (many CPUs these days have 2 or more different types of cores with different performance). Perhaps, we ran out of thermal budget and the CPU ran at a lower clock speed to avoid overheating? Or, since we are on a JVM, perhaps the compiler saw slightly different information for the types/behavior seen in the first and second iterations, and thus, made slightly different optimization decisions? This is possible because compilation happens on a background thread, and thus, even when the benchmark is deterministic, the used profiling information is to some degree racy.

It could also be something entirely different however, which means, we do not really know.

For designing your benchmark methodology, you’ll need to decide on how to take these variables into account. Sometimes, you may simply collect data from more runs, ideally many runs, so that you can characterize this behavior as part of the performance distribution one might likely observe in practice. In other cases, this might be too naive, and you need to carefully control for specific variables to get useful data from your experiments. This can include pinning threads to specific cores, fixing CPU frequencies, disabling address space layout randomization, etc. Each approach comes with different tradeoffs.

A Third Run, And More Data

We ran the same benchmark, with the very same setup, a third time. And, we actually had a...

Pitfalls of Benchmarking on Modern Systems

Related Articles

Incident with Github.com

(no title)

Scientists reverse brain aging, with a nasal spray

NHS apologises and admits Palantir have access to identifiable patient data

Cursor Origin