Modeling LLM Performance from First Principles
Home
Modeling LLM Performance from First Principles
2026-05-31
Tags:
Introduction
Over the past two years, I've taken a variety of courses on systems and high performance compute, and I've found myself particularly interested in LLM inference. However, over that same time frame, LLM Inference workloads have been evolving at an extremely rapid pace. This makes it challenging to internalize a robust model for workload performance. A lot of the knowledge on how to make LLM inference fast and effective is locked in the brains of a few talented engineers, and is shared as "performance tips" - general rules or heuristics that may not be applicable everywhere. If you've been interested in inference performance, you've probably seen some of these tips - things like: "batching is better for arithmetic intensity" or "batching is good for throughput but bad for latency". Industry heavyweights like Databricks publish guides such as this one to try and help out by specifying these rules, and they even quantify things such as throughput vs latency.
While these tips may be useful or correct, it's hard to develop a unified theory of mind as to why. Why are these suggestions helpful? Are they still accurate to modern workload dynamics? What are the underlying assumptions?
Those are the questions this post tries to answer. From first principles and basic environment variables available to every inference engineer, we will develop approximate algebraic expressions that help resolve what really affects our key metrics.
This post assumes some basic to intermediate understanding of the LLM workload. If you don't understand some of the following key terms, I recommend you keep Google (or your favorite LLM) nearby to ask some questions:
Prefill
Decode
KV Cache
Attention (the formula)
GPU
Memory Bandwidth
FLOPs
TLDR: We're going to do some math and figure out what really defines performance under various conditions for LLM Inference.
Key metrics
The key metrics we are going to focus on are:
Arithmetic Intensity (# Floating Point Operations / # Bytes loaded)
Throughput (Tokens / Sec)
Latency (Time per output token for a single user)
Why these three metrics? Read on!
Arithmetic Intensity
Arithmetic Intensity is the ratio of performed floating point operations to the number of bytes that you access from Memory.
$Arithmetic\ Intensity = \frac{FLOPs}{\#\ of\ Bytes\ Loaded}$
But why does this metric even matter? To figure this out, we have to look at a real piece of hardware. For the purpose of this post, we're going to look at an H100 SXM GPU
On this platform, we have 1,979 TFLOP/sec of BF16 performance, and 3.35TB/sec of memory bandwidth. Lets call these values $\pi$ and $\beta$.
Immediately, we can notice that our compute is a lot faster than our memory. In one second, we can process 1,979 trillion BF16 operations, but we can only actually load 1.675 trillion BF16 elements (3.35 / 2 because BF16 elements are two bytes).
Imagine the chip like an assembly line with one slow station and one fast station. If the fast station does one thing for every output of the slow station the fast stage - no matter how fast the workers there are - is forced to wait on the the slow stage, and end-to-end speed is constrained by the slow stage. However, if the fast station can do lots of things with an output of the slow station, the fast one doesn't have to wait as much - it can keep processing. This is the principle behind arithmetic intensity.
Arithmetic intensity simply represents the ratio of compute to memory access. For any given workload, one of the two factors (compute or memory) will be responsible for bounding the total performance. Given that compute is much faster, the ratio of the peak compute performance and the peak memory bandwidth (also known as the ridge point ) tells you how much compute you have to be doing for every byte you load such that the compute isn't waiting on the memory. More concretely, there's enough work to be done for each byte loaded such that the compute isn't finishing its work, and idling waiting for more bytes to load. For the H100 SXM, this ridge point value is an arithmetic intensity of ~590 FLOPs / byte (for BF16).
Roofline Model
The roofline model is a visualization of arithmetic intensity (Perf vs Arith. Intens.), governed by the following expression:
$Performance = min(\pi, \beta * Arithmetic\ Intensity)$
Why this expression? Think back to the previous section - if we are memory bound , that means our fast stage (compute) is waiting on our slow stage. We are memory bound if we're not doing enough work per every element we compute. To get the FLOPs/sec in this regime, we have to consider our memory bandwidth as well as the amount of work we're doing per byte, which is where the arithmetic intensity term comes from. Use the units as a hint - arithmetic intensity is FLOPs / byte, and memory bandwidth is bytes / sec. By taking the...