LLM Quantization Part 2.5: Do Floats Dream of Real Numbers? | LTT Labs
←BACK TO ARTICLES
LLM Quantization Part 2.5: Do Floats Dream of Real Numbers?
Utkarsh J.·
LLM Quantization Part 2.5: Do Floats Dream of Real Numbers?<br>LLM Quantization Part 2.5: Do Floats Dream of Real Numbers?
Utkarsh J.·
Table of Contents
Explore similarExplainerLLM (AI)
Surprise! Turns out there is quite a lot to talk about when it comes to LLMs (who knew?), so here is another article in our planned three part series. In Part 2 we discussed why we need so much VRAM to run LLMs and intentionally glossed over what BF16 or FP32 are exactly. Towards the end, we also briefly talked about storing every weight in one byte instead of two, so the storage that the model takes roughly halves. This is simple enough as an idea, but it is quite complicated to implement. We can't really understand what to throw away, or how far it can safely be thrown, until we can understand what one of those numbers actually is to a computer.<br>This turns out to be a stranger question than it sounds. When a model stores a weight like 0.1, it isn't storing 0.1. Floating point works in base 2, and just as 1/3 can't be written exactly in decimal (0.3333... forever), most decimal fractions can't be represented exactly in binary. 0.1 in binary, as explained below, becomes 0.000110011001100... repeating without reprieve, so the computer keeps the closest value it can fit in the bits it has and rounds the rest. This is why in almost every programming language 0.1 + 0.2 calculates to 0.30000000000000004 rather than 0.3. Nobody minds. You round it to 0.3 and get on with your life, because for almost everything you'd ever do with that number, the difference is beneath notice.
Models make the same bargain, at a scale of billions. Weights in models can be approximations like this, gesturing at a real number they will never quite reach. This article is about what a floating point is, how it decides which numbers it can and can't hold, and why understanding that is the whole key to shrinking a model without breaking it. It is time for some maths. Unlike Big Shaq’s Man’s Not Hot, the maths won’t be as quick but it will be as straightforward as scientific notation from school. Be advised, math-averse friends!<br>Note: This is Part 2.5 of 3 in our LLM Quantization series.<br>Part 1: What Even is an LLM?<br>Part 2: You’re Gonna Need a Bigger VRAMPart 2.5: Do Floats Dream of Real Numbers?
Part 3: Honey, I Shrunk the Numbers!<br>Thank you Bartowski (Colin Kealty) for his invaluable input.<br>What is a Floating Point?<br>Computers store numbers in binary: ones and zeros (most of them at least). The question is how many of those ones and zeros you allocate to represent a single number, and how you arrange them. This is what a data type defines.<br>Floating point is a specific method for representing real numbers (numbers with or without decimal points) in binary. A philosopher somewhere must be fuming at the mathematician’s choice to name it “real” when most of these numbers are infinite, unwritable, and unmeasurable. The "floating" part of the name refers to the fact that the decimal point isn't fixed. It can "float" to different positions, which lets the same number of bits represent an enormous range of values. It works by splitting the available bits into three fields, which will feel familiar if you remember scientific notation from school.<br>Take a number like the following:
Scientific notation has already split it into three parts:<br>Sign: Is the number positive or negative?<br>Exponent: The order of magnitude tells you how big or how small the number is, and therefore where the point floats to. This is the field that dictates the range of values the format can reach.<br>Significand, Fraction, or Mantissa: The significant digits themselves which set the precision of the number within that range.<br>The sign is easy enough to require only one bit in any format. The latter two parts are in constant tension. Spend more of your bit budget on the exponent and you can represent astronomically large and vanishingly small numbers, but more coarsely. Spend it on the mantissa and you get fine-grained precision, but over a narrower range. Every floating point format is just a different answer to "how many bits to use and how do we divide these bits?"<br>Bear in mind that the above example is in base 10. Computers don't have fingers, they only have transistors which can either be on [1] or off [0]. So floating point uses base 2 (powers of 2) instead, for the same reason binary exists in the first place: it maps directly onto the physical reality of the hardware. The first step is to convert the value from decimal to binary and use that for our mantissa and exponent. We’ll use the decimal (base 10) value of 6.5 as an example.
Concretely, a normalized floating point value is reconstructed like this:
The mantissa bits give you the digits, and the exponent slides the binary point to the correct magnitude. The bias is...