Holding the LLM Stack in Your Head — Nick Gustafson<br>Skip to contentThe ten arcs<br>The twelve arcs<br>01Mathematical & Computational Prerequisites<br>02Language Modeling Before Transformers<br>03Tokenization & the Input Pipeline<br>04Transformers from First Principles<br>05Decoding & the Real Inference Algorithm<br>06Inference Engines & Serving Systems<br>07Training & Post-Training<br>08Evaluation & Scientific Discipline<br>09Retrieval, Memory & Context Engineering<br>10Tools, Protocols & Agent Loops<br>expand all<br>Arc 01Mathematical & Computational Prerequisites8 posts01Vectors, Matrices, and the Spaces They Live In<br>Vectors as lists of activations, matrix multiplication as a linear map, and why every neural network operation bottoms out in matmuls.
02Norms, Dot Products, and Similarity<br>How cosine similarity, L2 distance, and projections work, and why they show up everywhere from attention scores to embedding retrieval.
03Distributions, Softmax, and the Chain Rule of Words<br>Softmax, categorical distributions, Bayes' rule, and the chain rule of probability — the four tools that make language modeling a well-defined math problem.
04Cross-Entropy, KL Divergence, and What Loss Functions Measure<br>Why cross-entropy is the standard LM loss, what it actually measures about two distributions, and how it connects to perplexity.
05Gradients and How Machines Learn<br>What a gradient is, why it points uphill, how backpropagation computes one efficiently via the chain rule, and what SGD does with it.
06Optimizers: Momentum, Adam, and Learning Rate Schedules<br>Why vanilla SGD is too slow, how Adam adapts per-parameter, and how warmup and cosine decay shape training dynamics.
07GPUs, Floating Point, and Why Precision Matters<br>IEEE 754, the difference between fp32/fp16/bfloat16, why mixed-precision training works, and the basics of GPU parallelism.
08A Short Prehistory of Statistical NLP<br>The arc from rule-based systems through statistical MT and log-linear models to neural approaches, giving you historical context for everything that follows.
Arc 02Language Modeling Before Transformers7 posts<br>Arc 03Tokenization & the Input Pipeline7 posts<br>Arc 04Transformers from First Principles9 posts<br>Arc 05Decoding & the Real Inference Algorithm9 posts<br>Arc 06Inference Engines & Serving Systems9 posts<br>Arc 07Training & Post-Training10 posts<br>Arc 08Evaluation & Scientific Discipline7 posts<br>Arc 09Retrieval, Memory & Context Engineering9 posts<br>Arc 10Tools, Protocols & Agent Loops9 posts
Don't know where to start?<br>If you want to understand attention<br>The minimum path to really seeing how a transformer layer works, not just reciting the shapes.<br>1Vectors, Matrices, and the Spaces They Live In<br>2Self-Attention: Q, K, V from First Principles<br>3Multi-Head Attention and Representation Subspaces
If you care about why inference is slow<br>Start from the one-new-row insight, then follow the KV cache and the systems built around it.<br>1Prefill vs. Decode: The Two Phases of Inference<br>2Why One New Token Means One New Row<br>3The KV Cache from First Principles<br>4PagedAttention: Virtual Memory for the KV Cache
If you're building with RAG<br>Just enough retrieval theory to make the engineering decisions actually make sense.<br>1Embeddings from Scratch: From Word2Vec to E5<br>2Chunking Strategies<br>3Rerankers and Cross-Encoders<br>4RAG Architectures End to End
If you're building agents<br>The loop, the protocol, the transcript formats, and what the model actually sees.<br>1A Short History of Agents: From ReAct to 2026<br>2Function Calling as Structured Generation<br>3The Agent Loop: Model, Runtime, Tool, Resume<br>4MCP: A Cross-System Standard for Tool Integration
The series is a full first draft. I'll be grinding through polish, corrections, and the odd rewrite. If you spot something wrong, contact info is here.