Tail Scale — Bill de hÓra

Open Menu<br>Close Menu

Bill de hÓra

Open Menu<br>Close Menu

Tail Scale

Jul 7

Written By Bill de hÓra

"Our p99 is fine" and "one user in six has a bad time" are the same sentence. We rarely hear it that way, and in this post I wanted to go through why that can be.<br>tail-scale is a small browser tool, a toy, for visualizing the tail-at-scale effect: the way a rare slow call stops being rare once a web page, a phone screen, or a service waits on enough of them. It's a single standalone HTML file running an in-browser Monte-Carlo simulation of serial and parallel fanout effects. You can download the page, open it in a browser and use the sliders to watch how good-put and bad-put numbers adjust. The model itself is captured in some reference python code that the web page is derived from.<br>The idea isn't mine, far from it. In 2013, Dean and Barroso named it in their paper, The Tail at Scale. In 2021, Marc Brooker laid out the bimodal model in a startlingly brief and imo classic blog post, Tail Latency Might Matter More Than You Think. Both put us in the same, uncomfortable place: a system can run inside its percentile and service level targets, while a real share of its users have a slow time in the aggregate. So credit where credit is due. Marc Brooker's is the bimodal model tail-scale uses, more or less unchanged. Dean and Barroso's paper is where the effect was named and where the hundred fan-out backend figure comes from that makes our 18 and 40 wide fan-outs we’ll see in the examples seem a bit less wild. I can’t recommend reading them enough.<br>Math will tear us apart again<br>Give every service call in our system a w chance of breaking a latency boundary. So, a p99 call is w = 1% . If a page load or app screen waits on N of them, and can only complete as the slowest call, the chance it stays quick is (1 − w)^N, and the chance it's slow is one minus that.<br>The tail-scale page has a row of dots, one per backend, each green while a call is inside its service target and red when outside. They flicker as the simulation runs, and every so often one returns a tail call and turns red. The result is marked slow whenever any single dot is red. A counter shows how the observed slow rate climbs toward the predicted 1 − (1 − w)^N slow rate.<br>Eighteen calls at p99 is 1 − 0.99¹⁸ , or ≈17 ` making for one result in six underperforming:

What catches us out isn't the math as much as we tend to reason about latency with an average in our heads and look at service quality on a per service basis, when the average is the wrong summary for the maximum of many draws, and many services contribute to the actual experience in modern internet products.<br>All systems green<br>Nothing about any backend is problematic on the face of things: each is still good ninety-nine times in a hundred. Their graphs will be green. Push to forty calls and that rare event ties up a third of the results:

The wide fan out case is brutal: a request touching a hundred backends, each working at p99 means only about 0.99¹⁰⁰ , or ≈37% of responses are good. A 10-30 fanout is absolutely not unheard of, and while numbers like 40 and 100 might seem excessive, both Google and Amazon have described systems in those ranges. And let’s remember this is happening per page and screen load, so in a session a user will have multiple opportunities to experience slowness.<br>it’s not so much that there’s a distance between ‘every service met its SLO’ and ‘the page was slow’, because both are true at once. More often than not no single team's dashboard monitors the composition of all the teams’ calls. In that sense per-service monitoring is built to show us the first and structurally doesn’t show us the second, because the second appears downstream in the end result.<br>At swim-two-failures<br>A fan-out and a serial chain fail differently. I got this wrong in my own first swing at the tool so I’m inclined to make a point of it. The bimodal split the tool uses, where parallel fan-out is a max, and a serial chain is a sum, is based on Marc’s post. A fan-out punishes you with variance, a chain with the mean.<br>A fan-out, waiting on the slowest return, means one unlucky call drags the whole result, so the tail thickens over time. A 1% event becomes more common as our N fanout grows in the 1 − (1 − w)^N slow case.<br>A dependent-chain sums unlucky calls which means they averaged away, and what breaks the target instead is the mean accumulating. Fifteen calls at a mere 80ms is 1.2 seconds, but that’s well past a 250ms target on the average alone, with no slow call required.

The declaration of interdependence<br>The (1 − w)^N mode assumes the calls are independent, but in practice they often aren't. A shared cache, a common lock, a GC pause on a service everything touches, a proxy server, a database write lock are all normal things we build on. When they stall or become contended, they impair the whole fan-out or head of line multiple serial calls. And so tail-scale adds a...