AI At Home Part 2: Multi GPU Drifting

No, We Have AI At Home

Chapter 2: Multi-GPU Drifting

August 20 2026

In the last chapter I built a ridiculous home server out of e-waste-grade GPUs to run AI language models. Here, I'll be talking about what I needed to do to squeeze some kind of reasonable performance out of this kind of setup. (This is going to be using existing code and techniques, messing around with llama.cpp settings and so on; writing new ROCm kernels is out of scope for this chapter).

I'm gonna go into some background here. If you already know how transformer models work, go ahead and jump to section 2. If you already know how multi-GPU parallelism works and just want to get to the part where I'm testing things, jump to section 3.

section 1: attention

Okay, basically all of the AI text generation software that's currently in use everywhere are instances of the "transformer model" or "large language model". The design and basic technique was introduced in the paper Attention is All You Need, which is probably the most important paper in the field of computer science in the past, I dunno, twenty years? The paper is pretty readable as far as these things go. I'm sure every software engineer reading this blog post has already read it, right? (Right?)

Anyway. I'm going to over-simplify things a bit here and focus from the perspective of somebody who is trying to get these things to run fast on crappy hardware, and not go deep into the tensor math or talk much about model training, because this blog post is already going to be way, way too long.

The LLM works in terms of "tokens". A token is basically a word fragment; instead of inputting and outputting individual letters it's more efficient to chop these up into sequences of letters and have the model process those. (This is why early AI models were bad at correctly answering questions like "how many times is the letter R in the word raspberry?") Different models tokenize language differently, you can think of this as like a frequency encoding. Each time a model generates another token, it'll actually generate a probability distribution and then randomly sample one from that distribution, because language works better that way than picking the exact most likely next thing every time.

The language model is a neural network that's divided into layers. You have an input layer and an output layer and a bunch of layers in between that don't directly interact with the input or output ("hidden layers"). The input layer takes in the entire input prompt, and then each layer does math on the output of the previous layer in series. The thing where the model looks at the entire input at once, and looks at the relations between different tokens at different points in the input series, is called the "attention mechanism". If you've been reading about AI language models you probably have heard somebody confidently claim "these AI models are just next word generators, like a Markov chain is", and then you probably noticed that these AI models generate very different outputs than a Markov chain does, and wondered where exactly that guy went wrong. Well, a Markov chain doesn't have the attention mechanism, it just generates a new token based on the previous token in the series.

So, to generate the next token, the model reads in the entire tokenized prompt, turns this into an embedding matrix (each token gets turned into a vector where the length is the hidden dimension of each layer), then does the attention math on each layer (gigantic matrix multiplication for each token, for each layer in series), samples a new token, adds it to the prompt, and keeps doing this in a loop until it gets to a token that indicates that it's time to stop.

fig 1 from "Attention Is All You Need" (Vaswani et. al., 2017)

For our purposes, what this means is that every time the computer generates a token, it needs to read in the existing context, and also every weight in the model, in order to do all those matrix multiplications to generate the token. I'd mentioned the Gemma4-31B model in the last chapter; as the name implies, the model has 31 billion weights (divided into 60 layers). We've got to load all of them into the GPU to calculate the next token. Loading these takes a lot longer than the actual attention math does; token generation is (usually) limited by memory bandwidth rather than compute. This is why the server I built has all of those GPUs with lots of VRAM attached to them; the model weights and KV cache need to be in VRAM that can get to the GPU quickly. The memory attached to the CPU is by comparison a lot slower. (This is also why we are in a memory shortage right now as the entire industry shifts to prioritize producing high-bandwidth memory for data center GPUs).

This obviously isn't going to scale super well. As model sizes increase, token generation slows way down; practical limits on this kind of thing got hit already. In response, we have the...