Exercises in benchmarking and evals, part 7: DeepSWE, Senior SWE-Bench, napkin math, and winter tires

This is part of a series of exercises on benchmarking, evals, and experimental design (1, 2, 3, 4, 5, 6)1. We're going to look at three questions, which are presented before the answers to give you time to think about the questions before seeing the answers.

29. A friend of mine is reviewing performance orders of magnitude to prep for computer performance interviews and found that https://github.com/sirupsen/napkin-math (5.4k stars) was the top hit. The README's tables include:

Napkin Math performance estimates

Operation<br>Latency<br>Throughput<br>1 MiB<br>1 GiB

Sequential Memory R/W (64 bytes)0.5 ns<br>├ Single Thread20 GiB/s50 μs50 ms<br>├ Threaded200 GiB/s5 μs5 ms<br>Network Same-Zone10 GiB/s100 μs100 ms<br>├ Inside VPC10 GiB/s100 μs100 ms<br>├ Outside VPC3 GiB/s300 μs300 ms<br>Hashing, not crypto-safe (64 bytes)10 ns5 GiB/s200 μs200 ms<br>Random Memory R/W (64 bytes)20 ns3 GiB/s300 μs300 ms<br>Fast Serialization [8] [9] †N/A1 GiB/s1 ms1s<br>Fast Deserialization [8] [9] †N/A1 GiB/s1 ms1s<br>System Call300 nsN/AN/AN/A<br>Hashing, crypto-safe (64 bytes)100 ns1 GiB/s1 ms1s<br>Sequential SSD read (8 KiB)1 μs8 GiB/s100 μs100 ms<br>Context Switch [1] [2]10 μsN/AN/AN/A<br>Sequential SSD write, -fsync (8KiB)2 μs3 GiB/s300 μs300 ms<br>TCP Echo Server (32 KiB)50 μs500 MiB/s2 ms2s<br>Random SSD Read (8 KiB)100 μs70 MiB/s15 ms15s<br>Decompression [11]N/A1 GiB/s1 ms1s<br>Compression [11]N/A500 MiB/s2 ms2s<br>Sorting (64-bit integers)N/A500 MiB/s2 ms2s<br>Proxy: Envoy/ProxySQL/Nginx/HAProxy50 μs???<br>Network within same region250 μs2 GiB/s500 μs500 ms<br>Premium network within zone/VPC250 μs25 GiB/s50 μs40 ms<br>Sequential SSD write, +fsync (8KiB)300 μs30 MiB/s30 ms30s<br>{MySQL, Memcached, Redis, ..} Query500 μs???<br>Serialization [8] [9] †N/A100 MiB/s10 ms10s<br>Deserialization [8] [9] †N/A100 MiB/s10 ms10s<br>Sequential HDD Read (8 KiB)10 ms250 MiB/s2 ms2s<br>Random HDD Read (8 KiB)10 ms0.7 MiB/s2 s30m<br>Blob Storage GET, if-not-match 30430 ms<br>Blob Storage GET, 1 conn (128KiB)80 ms100 MiB/s10 ms10s<br>Blob Storage GET, n conn (offsets)80 msNW limit<br>Blob Storage LIST100 ms<br>Blob Storage PUT, 1 conn (128KiB)200 ms100 MiB/s10 ms10s<br>Blob Storage PUT, n conn (multipart)200 msNW limit10 ms10s<br>Network between regions [6]Varies25 MiB/s40 ms40s<br>Network NA Central East25 ms25 MiB/s40 ms40s<br>Network NA Central West40 ms25 MiB/s40 ms40s<br>Network NA East West60 ms25 MiB/s40 ms40s<br>Network EU West NA East80 ms25 MiB/s40 ms40s<br>Network EU West NA Central100 ms25 MiB/s40 ms40s<br>Network NA West Singapore180 ms25 MiB/s40 ms40s<br>Network EU West Singapore160 ms25 MiB/s40 ms40s

Show full table

What's wrong with this benchmark?

30. I keep seeing people reference DeepSWE and Senior SWE-Bench to "prove" that their favorite model is better than other people's favorite models or just as generally good benchmarks, such as in

What's wrong with these benchmarks?

31. People frequently say that winter tires are superior to all-season tires in cold weather. For example, on googling "all season tires during winter cold" (no quotes), the Google AI summary leads with

All-season tires lose traction and stiffen in freezing winter temperatures. Their rubber compounds are designed for warmer weather and become hard below 7°C (45°F), leading to significantly longer braking distances and reduced grip ... The rubber in all-season tires cannot maintain pliability in sub-zero temperatures, causing them to perform more like hard plastic on snow and ice.

Given that there are a lot of internet comments in the training data, this is a reasonable comment, in that I frequently see variations on this comment on discussions of which tires one should use.

What's wrong with this benchmark?

29. Napkin math numbers

Random memory access latency

The thing that immediately jumped out to my friend (Jamie) as odd was random memory R/W listed as 20ns, since random memory R/W is implied to be a real DRAM read (as opposed to a cache hit), which he felt this should be around 100ns for an order of magnitude estimate.

As we were chatting about this, he noted that the README uses the term "latency" for some things that aren't really latencies. Then, when he pulled up the code for random memory read latency, he found the following (if you want another exercise, consider what's wrong with the following code before reading the explanation below):

while test.i

Jamie noted that there's no data dependency between the loop iterations, so the memory reads here happen in parallel. Since the alleged latency number is determined by finding the average time for an access, this is incorrect because the CPU can have multiple loads in flight at once. If you wanted to measure latency this way, you'd have to introduce a dependence between loads, to prevent overlapping accesses (we discussed a related topic in exercise 19, covered in part 4 of this series).

Random SSD read

I agree with all of Jamie's comments, although I didn't really flag the use of the term latency myself because...