What Is LLM Inference, Really? A Deep Technical Walkthrough - Karthika Raghavan

You are using an outdated browser. Please upgrade your browser to improve your experience.

What Is LLM Inference, Really? A Deep Technical Walkthrough

An Engineer’s annotated tour through what actually happens when you hit send — from bytes to tokens to embeddings to attention to the word your model finally spits out. No skipped steps. No “and then magic happens.”

Karthika Raghavan

Software Engineer • Distributed Systems • LLM Infrastructure

Follow

Vancouver, BC

GitHub

LinkedIn

Custom Social Profile Link

-->

1. What Is Inference?<br>2. The Artifact: What’s Actually in That 10GB Download?The “Manual” (The Header)<br>The “Hardware” (The Tensors)<br>Quantization: Shrinking the Brain

3. The Three Phases: A Map Before the Territory<br>4. Tokenization: Chopping Text Into NumbersIs Tokenization CPU-Bound?

5. Prefill: The Model Reads Your PromptThe Embedding Matrix<br>How Do the Model Weights Help Here?

6. Positional Embeddings: Teaching the Model About OrderHow Is It Calculated?<br>CPU Bottleneck in Prefill?

7. The Transformer Layers: Where the Real Work Happens<br>8. Decoding: One Token at a Time, ForeverDecode Step 1: Predicting “on”<br>Decode Step 2: Predicting “the”<br>The Sampling Step (Where Creativity Lives)

9. Why Memory Is the Decode Bottleneck<br>10. KV Cache: The Most Important Data Structure in InferencePrefix Caching: Free Speedups

11. Attention: The Mechanism That Makes It WorkSelf-Attention in Plain English<br>Paged Attention: Virtual Memory for KV Cache

12. Continuous Batching: The Throughput Unlock<br>13. The Metrics That MatterTTFT — Time to First Token<br>ITL — Inter-Token Latency (aka TPOT)<br>KV Cache Hit Ratio<br>KV Cache Usage<br>Scaling Strategy by Traffic Shape

14. Where Does the Time Actually Go?<br>15. The Inference Engines Worth KnowingOllama<br>vLLM<br>TGI (Text Generation Inference)<br>TensorRT-LLM<br>The META / Research Options

16. The Languages Behind It All<br>17. What CPU vs. Memory Intensive Means, Summarized<br>18. What an SRE Actually Needs to Know<br>The Summary: A Mental Model for ProductionThe “War Room” Reference Table<br>The Final Principles (Your TL;DR)

Conclusion: The Field Is Moving, The Fundamentals Aren’t

Let me be honest with you. When I started working on LLM infrastructure, I had eleven years of distributed systems experience. I knew Kafka, Kubernetes, Prometheus. I could debug a partition rebalance in my sleep.

And yet the first time someone asked me what actually happens during inference, I said something like “the model reads the prompt and generates tokens.” Which is technically true the same way “a database reads your query and returns rows” is technically true — accurate, useless, and deeply embarrassing for someone drawing a principal engineer’s salary.

This post is what I wish I’d had on day one. We’re going to walk through the entire inference pipeline — from the moment your request arrives to the moment you see text on screen — with real examples, honest explanations of where the performance goes, and enough detail that you can actually reason about production problems.

No “and then the transformer does its thing.” No skipped steps. Strap in.

1. What Is Inference?

Training is where you take a massive dataset, run it through a model millions of times, and slowly adjust billions of numerical weights until the model gets good at predicting the next word. Training is done once (or occasionally). It costs millions of dollars in GPU-hours and requires a team of researchers.

Inference is what happens afterward, every time someone uses the model. It’s the model using those learned weights to respond to new input. No weights change. No learning happens. It’s pure forward-pass computation.

Think of it like this: training is baking the bread. Inference is slicing it and serving it to customers. The bread (weights) is done. The kitchen (inference engine) just has to plate it fast enough that the queue doesn’t back up to the street.

The inference engine is the runtime that takes the frozen model weights and executes them against an input. The same weights can run on Ollama, vLLM, TensorRT-LLM, or TGI — and get meaningfully different performance from each. The weights don’t change. The execution strategy does.

This distinction matters operationally: inference is not a solved problem . Serving a model efficiently at scale is a full engineering discipline.

2. The Artifact: What’s Actually in That 10GB Download?

When you run ollama pull mistral or grab a model from HuggingFace, you aren’t just downloading a “program.” You’re downloading a massive, frozen brain in a box. If you’ve ever wondered why a model that “just chats” takes up 10GB of your SSD, it’s because it is packed with billions of tiny numerical “preferences” the model learned during its training phase.

Think of the GGUF (or Safetensors ) file as a giant Ikea flat-pack box. To build the working model, you need two things: the Instruction Manual and the Hardware...