NVLink, NVSwitch, and all that | Doubleword<br>I wrote before about<br>‘scale-out’ fabrics — InfiniBand,<br>RoCE, and more<br>generally<br>RDMA — the networks and paradigms that stitch thousands of machines into a datacenter<br>island.

This post is about the other kind. The names come from an old argument about<br>servers: when you need more computer, you can either buy more machines and<br>scale out, or buy a bigger machine and scale up. A scale-up fabric is an<br>attempt to connect processors so tightly that they can behave as one. Inside a<br>modern (NVIDIA) GPU machine the fabric is<br>NVLinkThe terms are a bit frustrating really. Because you can scale up your LLM<br>inference domain over your scale-out network. And you can certainly scale out inside a<br>scale-up domain..

Fast, or far

Let’s follow the physics of sending bytes over a link. At the lowest level: an<br>NVLink is a bundle of wires, set up as differential<br>pairs. Each pair is<br>driven by a SerDes: a circuit that<br>turns wide, slow on-chip data into one very fast signal on the wire.

The signal on the wire is just a voltage changing over time. To turn it back<br>into zeros and ones the receiver has to sample it at the right instants. Before<br>deciding what to put on the wire, we have to decide where the receiver’s sense<br>of time is going to come from.

Length-matching meanders next to a BGA package<br>(source).

One answer is to send a clock pulse, separate to the data, on its own wire, and<br>sample as the clock ticks (a forwarded clock). This is how DDR<br>memory and<br>HBM work. But it only<br>works if the clock’s ticks arrive close enough to all the signal bits for the<br>receiver to sample them cleanly. The receiver can deskew small differences, but<br>every data wire still needs about the same delay and, in practice, about the<br>same length. This is why PCBs are covered in<br>squiggles. At a hundred gigabits per second, one bit occupies about ten<br>picoseconds: a millimetre or two of trace. That kind of matching is possible<br>across centimetres, and gets much harder across metres.

We can instead synchronize each wire individually, sending its clock over each<br>signal wire itself. We know that every transition in the signal (each 0-to-1 or<br>1-to-0) marks a bit boundary. So we can use a circuit at the receiver (clock<br>and data recovery) that keeps a<br>local oscillator locked to those transitions. The problem is that it works only<br>if the transitions keep coming. If you send a million zeros in a row, the<br>oscillator can start to drift. To keep the data transitioningStatistically, not provably. In principle you can craft data that XORs to<br>a long run of zeros ("killer packets", seen in<br>SONET). Modern<br>links use long scrambler polynomials to make this impractical., we XOR it<br>with a pseudo-random sequence that both ends can generate<br>(scrambling).

Top: each edge in the data re-centres the receiver's clock, but across<br>the run of zeros it free-runs (drawn a few percent fast) and the samples slide<br>off-centre until one lands in the wrong bit. Bottom: the same payload<br>scrambled, the edges never stop.

RAW

✗ wrong bit

SCRAMBLED<br>same payload ⊕ PRBS

RAW

✗ wrong bit

SCRAMBLED<br>payload ⊕ PRBS

Once we can carry a timing signal, we need to put our 1s and 0s on the wire.<br>One of the simplest ways is: high<br>voltage for a 1, low for a 0. That scheme is called<br>NRZNon-return-to-zero: return-to-zero being a version where you return to<br>zero between every signalled bit.

To signal faster on a single wire under NRZ you have to tick faster. A trace on<br>a board works like a low-pass<br>filter: the copper, and the<br>insulation around it, both absorb more at higher frequenciesThe copper loses because high-frequency current crowds into a thin shell<br>at the conductor's surface (the skin<br>effect). The insulation loses<br>because the signal's energy travels through it via the electromagnetic field,<br>and the material's molecules absorb some of it as the field alternates<br>(dielectric<br>loss)., adding<br>attenuation that the receiver has to make back. It does this with filters that<br>re-boost the high<br>end, but this boosting amplifies the high-frequency noise along with the<br>signal.

Rather than keep paying that rising attenuation, at some point it’s better to<br>try to send more bits per tick.<br>PAM4 signals four<br>voltage levels instead of two, so each tick carries 2 bits of information. The<br>cost is that the levels sit a third as far apart. As the level spacing gets<br>closer, errors in decoding start to become more likely.

PAM4 links aim for roughly one bit in ten thousand wrong before correction,<br>where NRZ links without correction can aim for better than one in a trillion.<br>At these speeds an error rate of one in<br>a trillion means an error every ten seconds or so, which is rare enough you can<br>manage it at the link layer with checksums and resends. One in ten thousand<br>means millions of errors a second, which is far too many to handle by resending. So<br>PAM4 is usually paired with forward error<br>correction. This costs<br>latency, since the decoder has to collect a whole block before it can...