When Bursty Traffic Makes LLM Inference Faster - Harvard Systems Group

TL;DR: Burstier arrivals made LLM inference faster in my benchmarks, contradicting standard intuition—burstiness separates decode tokens from heavy prefills, reducing interference. However, that interference is largely attributable to a kernel optimization artifact, and the benefits of burstiness only appear when mixed batches are inefficient.<br>Introduction<br>Bursty workloads have been giving computer scientists headaches for decades, and modern LLM serving is no different. Big bursts create instantaneous load and latency spikes, and there&rsquo;s a whole host of literature exploring how to deal with them.<br>The reason burstiness hurts, intuitively, is because we have the same mean arrival rate but requests are more likely to arrive very close together, or very far apart. To the server, it looks like it&rsquo;s getting flooded within a short window, followed by a dead period. The result is worse performance but the same long run utilization compared to a more regular workload.<br>So, when I was doing some routine vLLM benchmarking and found that as the arrivals got more bursty, latency decreased rather than increased, it was pretty surprising! Beyond contradicting standard intuition, this would have implications for how request routing should be done in production inference clusters.<br>In this short blog, we&rsquo;ll explore the data that gives confidence to those results, but surfaces a confound that takes the project from the best thing since sliced bread to&mldr; the second best thing since sliced bread.<br>Results: burstiness lowers latency<br>In this plot, we see that as the coefficient of variation of the interarrival times increases (more bursty), the median time per output token (TPOT) and end-to-end latency (E2EL) decrease. This was observed on both synthetic and realistic workloads, like ShareGPT and BurstGPT. Time to first token (TTFT) is much more config dependent, though we&rsquo;d expect it to worsen due to the &ldquo;clustering&rdquo; of arrivals.<br>Figure 1: Median/p90 TPOT and E2EL decrease ~50% between CV = 1 (Poisson process) and CV = 3.16 (very bursty arrivals) on BurstGPT. TTFT improves non-monotonically. Mean TPOT (not shown) improves 57% from 70 ms to 30 ms. Bands represent standard deviation across 6 replications at 1k requests.<br>In fact, burstiness generally helps most of the distribution, sometimes slightly hurting the tail (Fig. 2).<br>Figure 2: TPOT estimated CDF on BurstGPT shows that burstier arrivals improve the whole distribution.<br>The appendix includes far more results, spanning multiple models, GPUs, and datasets, increasing our confidence that this behavior generalizes.<br>A familiar explanation<br>Loosely, higher burstiness means the gaps between arrivals are more weighted to very small or very large values. We can then picture getting a clump of arrivals, followed by a quiet period where the engine performs a bunch of decode without interruption. It follows that we should expect more iterations which are exclusively decode, but more importantly, that decode tokens get separated from large prefills.<br>This hypothesis is supported by the data, which follows the trend shown in Fig. 3: higher burstiness increases the fraction of tokens that are part of decode-only iterations.<br>Figure 3: The fraction of output tokens in a decode-only iteration increases by 16.8 percentage points at high burstiness. (Qwen3.5-9B | GH200 | BurstGPT)<br>Additionally, Fig. 4 confirms that prefills get compressed together, since we see that at high burstiness, mixed batches get larger and more prefill-heavy.<br>Figure 4: Higher burstiness concentrates arrivals, leading to larger and prefill-dominant mixed batches. (Qwen3.5-9B | GH200 | BurstGPT)<br>Note that figures 1 through 4 use Qwen3.5-9B, so the effects there are GDN-related and not tied to the Flash Attention versioning discussed later.<br>It&rsquo;s widely accepted that mixing prefill and decode causes interference, hence slower iterations. This is one motivator behind PD disagg. It follows that if we create more exclusive decode iterations, we reduce the prefill-decode interference, making the average decode token cheaper and speeding up the engine.<br>Using these few numbers, we can also loosely recover the 57% TPOT reduction presented above.<br>Show the derivation (optional)At low burstiness, 71.2% of generated tokens experience small decode-only iterations averaging 26.05 ms, while 28.8% experience large mixed iterations averaging 156.98 ms. This gives an expected GPU-forward interval of 63.73 ms. At high burstiness, the corresponding mixture is 88% at 14.36 ms and 12% at 116.02 ms, giving 26.59 ms—a 58.3% reduction, close to the observed 57% mean TPOT improvement.<br>Additionally, while it may seem counter-intuitive that the 3084 token mixed batch is faster than the 2074 token one, note that it contains fewer decode tokens, which are more expensive than prefill tokens. This is shown in the linear fit of Fig. A1...