What is Inference? | Internals for Interns<br>Welcome to a new series! For most developers today, using a large language model means one thing: an HTTP call to somebody else’s computer. You send a prompt to an API, tokens come back, and everything in between is somebody else’s magic.<br>But here’s what I find much more interesting: you can run these models locally, inside your own process, on your own hardware — and if you’re a Go developer, you can do it directly from Go. Two projects make this genuinely pleasant today: Yzma , which lets Go call the llama.cpp libraries directly (without cgo — that gets its own article later in the series), and Kronk , which builds a high-level, OpenAI-API-feeling SDK and model server on top of Yzma. And underneath both of them sits llama.cpp , the C/C++ inference engine that made running LLMs on ordinary hardware practical.<br>In this series we’re going to understand that whole stack from the inside: what llama.cpp actually does when it “runs a model,” how Yzma manages to call it from Go, and how Kronk turns all of that into an engine you’d actually want in production.<br>📌 A note on scope<br>The goal of this article is to understand the mechanics around the model — how a model file is stored, loaded, and run — not what happens inside the neural network itself. The math inside those layers is a whole topic of its own, and it’s out of scope here.<br>The llama.cpp details are against master as of July 2026 (commit ad8d8219) — llama.cpp moves fast and doesn’t do stable releases, so I’m pinning a commit instead of a version. And this is deliberately a conceptual tour: we’re not going to get into the C++ code details. We’ll also stay out of the Go layers entirely — Yzma and Kronk get their own articles.
With the scope set, let’s dive in.<br>What Inference Actually Is<br>Let’s start by demystifying the word. A language model has two very different lives.<br>The first life is training : you show the model terabytes of text, and an optimization process gradually adjusts billions of numbers — the weights (also called parameters ) — until the model gets good at one narrow task: given a sequence of text, predict what comes next. Training takes months and data centers. We’re not talking about training in this series, at all.<br>The second life is inference : training is over, the weights are frozen, and now you just… use them. The best mental model is a plain, pure function: it takes two inputs — your sequence of text and those billions of frozen weights — and returns one output, a prediction for the next token. Something like nextToken(input, weights). That’s it. That’s the whole secret. No learning happens, nothing is updated — the weights are read-only arguments, so a model file is exactly as smart the millionth time you call it as the first.<br>📌 A note on weights<br>If you’re curious about how much memory those weights actually take up — how quantization shrinks them, and why they usually dominate the VRAM you need to run a model locally — Kronk’s VRAM calculator write-up is a nice read.
So when you download a “model,” what you’re downloading is essentially a giant pile of frozen numbers — those weights — plus a description of how to use them. And “running” the model means doing a very large amount of arithmetic between your input and those numbers. If that sounds suspiciously hand-wavy, don’t worry: we’ll demystify the weights later in the article, when we crack open an actual model file. For now, all you need to hold on to is that a model is a huge pile of numbers that somebody else already calculated for you.<br>We’ve already mentioned tokens a couple of times now — but what exactly is a token? Let’s see.<br>Tokens: the model’s alphabet<br>There’s one catch before any arithmetic can happen: models don’t read text. They work with tokens — integer IDs from a fixed vocabulary, where each ID corresponds to a chunk of text (a word, a piece of a word, a punctuation mark).<br>Our prompt “The capital of France is " might become something like five or six token IDs — say [791, 6864, 315, 9822, 374] (the exact IDs depend entirely on the model’s vocabulary). From this point on, the model never sees letters again; everything downstream is done in terms of these integers.<br>So the model receives a list of integers. What does it actually do with them?<br>One pass, one prediction<br>Here’s the shape of the computation, stripped to its essence. One important thing up front: the model doesn’t process the input word by word in separate rounds — the whole sequence of tokens goes through the network together, in a single pass, and out comes a single prediction. Let’s walk through it with this diagram:
Reading it left to right, we start by converting each token ID into an embedding — a vector of numbers that represents the...