Count the bytes, not the FLOPs · thepragmaticquant.com

Skip to content<br>Count the bytes, not the FLOPs<br>Sankalp Gilda · July 11, 2026 · tsbootstrap · performance · memory · engineering · 5,982 words · ~30 min<br>Contents 01 Wall one: memory traffic<br>02 One pass, and nothing left behind<br>03 Wall two: the state you carry<br>04 Two optimizations we measured and deleted<br>05 The scoreboard, losses included<br>TL;DR: tsbootstrap’s fast path used to build every bootstrap replicate into one giant batched tensor: on a routine job, a 160 MB tensor pushed out to main memory and hauled back to produce one summary number per replicate. We rebuilt the hot path around two rules: never materialize an array whose only consumer is a reduction, and never carry state you can derive. Every head-to-head benchmark cell now favors the fused path; the settled benchmark grid sustains 3.1x to 20x on the longer series. The last stubborn cell, where the runtime was mostly Python building random-number seeds, dropped from 13.1 ms to 0.55 ms in the redesign’s validation run once we deleted the seed objects.

The full replicate tensor is still available when you want it, and on that materializing path our default backend remains slower than the incumbent. That row is printed below with the rest.

Early this summer, in the middle of a performance push I thought was going well, we ran tsbootstrap’s block-bootstrap engine head-to-head against arch, a mature econometrics library whose bootstrap loop has had years of polish, for the first time. We lost. At a realistic workload, a few thousand replicates of a few-thousand-point series, the whole call came back several times slower end-to-end, on the machine we developed on, before the compiled engine this article describes existed. Every release we had ever shipped carried that loss; users hitting exactly that workload had been paying it the whole time, and nothing in our tooling would ever have told us.

I had not seen it coming: our benchmark suite had only ever compared the library against its own history. A regression suite tells you when you get slower than yesterday; it cannot tell you that you ship slower than the other library, at the exact workload where users would notice. The day a head-to-head existed, the loss appeared. It had been there all along.

The profiler’s first answer was the ordinary one. Most of the measured deficit was removable interpreter tax: a per-replicate Python index loop, thousands of seed objects spawned per call, stacked object wrapping. But beneath it sat a second, structural problem, the reason the reflexive fix could not win either. The reflexive fix for Python overhead is to batch: replace the loop with a single vectorized NumPy operation. Batching removes the interpreter from the inner loop and swaps it for B-fold traffic through main memory (B being the number of bootstrap replicates, the count every cost in this story scales with) because the batched design materializes every replicate at once. The incumbent’s replicate loop is ordinary Python too. It wins by keeping its working set (the data its loop actively touches) small enough to live in cache, consuming each resample the moment it exists. We were not going to beat that by batching harder.

Here is the machine’s side of the argument. A CPU core computes only on what it holds in its registers. Think of that as your hands. Behind them sit the caches: L1, a desk you can reach without looking up; L2, the shelf behind you; L3, shared with the other cores, a cabinet down the hall. Behind all of that is DRAM: main memory, the warehouse across the street, bigger and slower at every step, orders of magnitude in latency top to bottom. The hardware hides none of that from your runtime; it hides it only from your source code.

Now put a bootstrap on that machine. A bootstrap, one time for anyone who has not run one: it re-draws your data thousands of times to see how much a statistic would wobble across plausible alternate samples, and each redrawn copy is a replicate. Take a modest, entirely realistic job: a series of n = 10,000 observations, B = 2,000 bootstrap replicates, float64 (8 bytes per number). Build all the replicates at once, the way a batched NumPy design wants to, and you have asked for a B x n tensor: 10,000 x 2,000 x 8 bytes = 160 MB . That is far beyond the tens of megabytes of L3 a core complex sees. Every byte of that tensor is written out to main memory, then hauled back in so the statistic can read it once, and if the statistic is a mean, the answer you keep is 2,000 numbers. Sixteen kilobytes, bought with 160 MB of round trips.

main memory (DRAM) cache boundary your data n = 10,000 · float64 replicate tensor 8 MB 40 MB 80 MB 120 MB 160 MB 16 kB one statistic per replicate: (B, d) Every replicate is a copy of your data.<br>The tensor is B times larger than the data. It lives in DRAM.<br>Gather and reduce in the same pass. Keep only the answer.

20 MB where 1.94 GB stood, B = 50,000.<br>measured 2026-07-04 on...