The mean means nothing | Farid Zakaria’s Blog

Skip to content

I was recently trying to validate some performance improvements related to lld at $DAYJOB and it was a little frustrating to see the improvements in our benchmarks but not in the live-production dashboards.

Having come from a background working on web-services, I was used to looking at individual time-series dashboards, sometimes over a few percentiles, and I was expecing to see some noticeable change but the data seemed too noisy to make any conclusions.

Turns out a colleague had also faced similar issues when trying to evaluate build-speed improvements. There are lots of variables that can affect the build: cold-cache, incremental, local, remote, etc. and the build times can vary wildly depending on the state of the system and the workload. She ended up leveraging a cummulative distribution function (CDF) to visualize the data and it was a revelation to me.

This led me to explore a few other different ways to visualize data, in addition to the CDF, and how a single image or statistic is often not enough to tell the whole story. This post will walk through a single synthetic dataset and show how different visualizations can tell different stories about the same data. The goal is to convince you to look at your data and not just summarize it with a single number.

Everything below comes from one synthetic dataset with a fixed seed. The<br>full script can be found in this gist. It is a single file with a<br>nix-shell shebang, so you can reproduce<br>every figure exactly as long as you are using nix.

Note<br>I leveraged AI to help generate the data and charts in this post for the story.<br>If that bugs you, sorry. 🤷

§The rollout that “made it worse”

Here is the setup: we operate a typical web-service and we rolled out a new caching tier over a week, hoping to cut request latency.

The change is fully deployed, and the latency dashboard that plots the mean looks like this:

Mean latency went up, from 112 ms to 122 ms. ☹️

A SEV is cut, we revert the change and write the postmortem. Right? 🤔

§One number, four stories

It’s often good practice especially for web-services to look at various percentiles, especially the tail end of the distribution like the p95 and p99.

statistic<br>before<br>after<br>change

mean<br>112 ms<br>122 ms<br>+9%

p50 (median)<br>99 ms<br>54 ms<br>−46%

p95<br>224 ms<br>454 ms<br>+103%

p99<br>309 ms<br>678 ms<br>+119%

Now we have a problem, and the problem is that everyone is right. The mean says<br>the change is a mild regression. The median (p50), says the<br>change is a big-win, the typical request got nearly twice as fast . The p99<br>says it’s a SEV, the worst requests more than doubling.

The mean and the median, computed from the very same numbers, point in opposite<br>directions.

Engineers are often taught to be data-oriented, but often it’s easy to cherry-pick the statistic that supports your argument.

§Look at the shape

The next basic thing you can do with a distribution is plot its shape.<br>Here are the two latency distributions, before and after, as densities:

There it is. 🤓☝️ The “before” is one tidy hump . The “after” is two humps .

This already explains the earlier contradiction but it’s a bit tricky to visualize correctly. The shape depends on a smoothing parameter we chose, the two fills muddy each other where they overlap, and it’s genuinely hard to read a percentile off it. I can see there are two populations; I can’t easily see where the median went.

§The best chart you’re not using

The cumulative distribution function (CDF) answers one question for every percentile at once: what fraction<br>of requests came in at or below x milliseconds?

CDFs are an extremely easy way to visualize multiple percentiles in a single chart. Depending on the curve, we can understand how the request latency is distributed across the entire population.

I found it incredibly useful to then plot the “before” and “after” CDFs on the same chart to see how they compare. You can then visualize the shifts at various percentiles and understand how the change affected the entire population.

In our story, the “after” curve has shifted left for request latencies below 140ms. That means more requests are finishing faster than before. The “after” curve is higher than the “before” curve to the right of 140ms which means more requests are finishing slower than before. The two curves cross at ~140ms which is the tipping point where the change goes from being a win to a loss.

Tip<br>Two CDFs that cross are the unmistakable signature of a change that no single percentile can summarize, because the sign of the effect depends on which percentile you ask.

§Who won, and by how much

The CDF tells us that the effect changes sign: faster or slower. The obvious next question is by<br>how much, at each point in the distribution. We can plot for<br>every percentile p, the after-latency minus the before-latency, known as the<br>shift function .

Below the zero line the change is faster; above it, slower.<br>We can...