Simulating cosmic rays to lobotomize LLMs — Benedikt Holm

Forty-nine billion bits of our precious intelligence can't add two numbers after one cosmic ray.<br>In space, there is a scourge known as cosmic rays. When one strikes the bits our computers are made of, they can spontaneously flip from 0 to 1, or vice versa. These bit-flips are mostly known for allegedly helping speedrunners win faster, meddlin' with Belgian politics, making Voyager-2 go out of sync, but they've also been shown to brutalize machine learning models. Today, people like Elon Musk plan to put data centers in space, where bit-flips are caused not just by cosmic rays but also by much stronger ambient radiation, and are thus far more common than at sea level. This got me wondering: how vulnerable are LLMs to cosmic ray attacks? But I can already hear you frantically typing "You idiot, there is such a thing as ECC, heard about it?", and yes I have, stop interrupting me, I'll get to it.<br>To preface, this is less a scientific writeup and more a fun little log of a neat experiment I came up with while putting my newborn to sleep. Nor is it entirely novel, since prior research has looked into bit-corruption of LLMs, though as far as I could tell it mostly focuses on targeted or adversarial bit manipulation. I also want to be clear that what I am simulating here is not particle physics, a GPU or a true radiation environment here; I'm uniformly randomly flipping model bits as a toy model of radiation-induced soft errors.<br>A lot of the work described here could be carried much further, but I find myself constrained both on time (newborn) and on compute, so this work is by necessity limited. Either way, I hope that you derive some enjoyment out of the horror I put these models through.<br>Me and Qwen during the experiment.<br>To run my experiment on how hard a cosmic lobotomization an LLM could handle, the first step I had to take was to select the subject. After considering my options, I initially opted to perform the experiment locally rather than rent equipment. This meant two things: 1) I could claim that I was performing important research for the local-LLM community (shoutout local.ai), and 2) I wouldn't have to spend too much money on expensive, out-biddable hardware on sites like vast.ai. This meant however that I was limited to models that could fit on my laptop's 5070 (yes, I am compute-poor). The victim that I ended up choosing was Qwen2.5-Coder-3B, a coding LLM trained to solve programming tasks. A thing to note, is that I ran Qwen in FP16, each weight being a 16 bit floating point number. Of the 16 bits, number 14 is the most magical, since it is the most significant bit (MSB) of the weight's exponent. A flipped bit in this position can change its number dramatically, as you can play around with in the following digital fidget toy:

= 0.02063reset0010010101001000

Click any bit to flip it. Bit 14 (red ring) is the cosmic ray: it turns 0.021 into 1352. Press reset to restore.<br>To measure how flipping bits around in the model's brain affected its outputs, I used the human-eval (HE) benchmark, a set of 164 coding problems. Note that HE is widely considered obsolete due to many models including it in their training data. I figured that since I was going to be performing low-accuracy brain surgery with a simulated space laser on Qwen, it didn't matter too much what it was trained on. Before starting the torture, I recorded the base model solving 139 out of the 164 HE problems, or about 85%. Remember this score, because it's not getting any better.<br>After getting the Qwen running, addressing the weights in memory was relatively straightforward, the Transformers library let me address individual weights directly, so flipping a bit in a given weight was as easy as XOR-ing it as so:<br>weight ^= (1<br>where x is the position of the bit to flip within the weight. This also made reverting the damage very quick and easy. In this experiment, the model inference was kept greedy, by setting the do_sample flag in the generation to False. This basically means that the seed or temperature plays no role in the inference, turning the output into a deterministic function of the value of the weights. And so it would have, if I had not varied batch sizes between runs to try to optimize my tps, resulting in slight wobbliness in the model's baselines between some runs, just in case you noticed that some runs start at slightly different HE baselines. If you didn't notice, please forget this last sentence.<br>Before starting, I was fairly sure that zapping the model bit-by-bit would take forever, so I designed a kind-of bit-flip tape, a seeded sequence of bit locations to flip. This let me precompute which bits would be corrupted in what order, and instead of flipping one bit and regenerating tokens each time, I could corrupt 500,000 at once and check for degradation in the HE score. If degraded, I knew the killing blow was somewhere in those half a million bits, so I could roll...