LLM Quantization Part 2: You’re Gonna Need a Bigger VRAM | LTT Labs
←BACK TO ARTICLES
LLM Quantization Part 2: You’re Gonna Need a Bigger VRAM
Utkarsh J.·
LLM Quantization Part 2: You’re Gonna Need a Bigger VRAM<br>LLM Quantization Part 2: You’re Gonna Need a Bigger VRAM
Utkarsh J.·
Table of Contents
Explore similarExplainerLLM (AI)Investigative
If you made it through the first article of this series, you now know that a large language model (LLM) is just a very large list of numbers, and that those numbers need to live somewhere when you're running inference (asking it to do something). More precisely, they need to live somewhere fast and physically close to whatever is doing the math, so the processor spends its time computing instead of sitting idle waiting for data to arrive.<br>If that processor is your GPU, which is usually the fastest hardware in the system for the job, that somewhere is Video RAM (VRAM), the dedicated memory on the graphics card itself. You can also run inference on your CPU, in which case the numbers just need to fit in ordinary system RAM, CPUs typically aren’t nearly as fast for operating LLMs. Either way, that memory fills up faster than you might expect, and this article is about exactly why.<br>Note: This is Part 2 of 3 in our LLM Quantization series.<br>Part 1: What Even is an LLM?<br>Part 2: You’re Gonna Need a Bigger VRAM<br>Part 3: Honey, I Shrunk the Numbers (coming soon)<br>Thank you Bartowski (Colin Kealty) for his invaluable input.<br>So Where Does All The VRAM Go?<br>It helps to think of VRAM as a desk. Four things compete for the surface when you are, let’s say, assembling something: the reference manuals you keep open the whole time (the weights), the running stack of notes that you might take to keep track of the parts (the KV cache), the scratch paper for some measurements you need during a step (the activations), and the baseline clutter of the lamp and your own elbows before you've started (overhead). The naive calculation of the VRAM capacity only accounts for the manuals. The real sum is closer to:
Let us take a look at all four components in more detail.<br>Weights<br>Recall from the previous article that every parameter (weight) is a dial, and the model is nothing more than the full set of those dials and their settings. Storing the weights is just the cost of writing down where each dial sits. That cost is the parameter count multiplied by the bytes used to record each one: an 8B model at BF16 precision (2 bytes per dial/parameter) is 16 GB, and it's a fixed cost. Meaning it is 16 GB whether your prompt is one word or one Dostoyevsky novel. The settings don't change as you use the model, so neither does this number.<br>BF16 or “bfloat16” is a format developed by Google Brain (hence the “b”). The 16 indicates that it uses a fixed 16 bits or two bytes (8 bits in a byte). This "fixed" storage cost is the easiest to account for when calculating VRAM, and is usually listed wherever you download your models from. The other three factors, however, are quite variable.
The multiple options for Gemma 4 with different parameter and quantization options on Ollama library<br>Source: https://ollama.com/library/gemma4
KV Cache<br>As we saw in Part 1, generating tokens is an iterative process: the model creates a token, adds it to the sequence, and repeats. To visualize this, imagine the model as a skyscraper where each floor represents a layer. A token is like a visitor entering at the ground level who must ascend every floor, undergoing processing at each stage, until reaching the roof as a next-token prediction. Sequential generation involves sending a stream of these visitors up the building, one after another.<br>Each floor has an administrator (the layer's attention mechanism) who interviews every visitor with three specific questions:<br>Query: Which previous visitors are relevant to me?<br>Key: How will future visitors identify that I am relevant to them?<br>Value: What information do I provide if a match is found?
A simplified flowchart of what the attention mechanism does on each layer of the model
For example, take the word "blue". Its Query looks back over the earlier visitors for anything that modifies a colour, and when it finds "dark," it pulls that context into its own running representation, so "blue" is now understood specifically as dark blue. The token itself is still "blue"; what changed is the meaning attached to it. Its Key and Value are what it leaves behind for later words: the Key advertises that it is a color, and the Value carries what it contributes if something matches. When "car" arrives, its own Query looks back, finds the "blue" Key, and pulls in the "blue" Value, so "car" is now contextualized as a blue car. Attention just lets each token gather the context of the ones before it.<br>Crucially, Queries only look backward at existing visitors. Once a visitor records its Query response, its Key and Value cards are fixed. The administrator never needs to re-interview visitors who have...