How prompt caching works - Paged Attention and Automatic Prefix Caching plus practical tips – sankalp's blog
Thanks for reading! ☕
How prompt caching works - Paged Attention and Automatic Prefix Caching plus practical tips
30 Nov, 2025
Table of Contents
Intro Lore and Motivation - Yapping about why I wrote this post and giving a brief on territory we are about to venture in
Tips to hit prompt cache more consistently - Why prompt caching matters and how to improve cache hits
LLM inference basics - Prefill, decode, and KV caching fundamentals
The memory problem - Traditional KV cache allocation challenges and why it doesn't scale
Paged attention - vLLM's OS-inspired solution with blocks and block tables
Prefix caching - Block hashing, longest cache hit, and the full picture
Prerequisite: Sections 2 onwards assumes familiarity with self-attention in decoder transformers. Refer nanoGPT or 3blue1brown.
Intro Lore and Motivation<br>Recently at work, I had to build a feature on a tight deadline. It involved chat plus tool calling components. I didn't give much thought to prompt caching as I was just trying to ship v0.
Following next week I started to optimise it and started realising some silly mistakes I had made under pressure. I ended up adding long user-specific data at the end of system prompt thinking that I just need to keep the longest prefix stable for a single conversation / messages array.
A messages array would look like
0. [system prompt + tool definitions]
1. user: what's up. please build this feature for me<br>2. assistant: can you tell me where to look, it's a big codebase<br>3. user: look into kv_caching folder<br>4. assistant: you're absolutely right! i will look there<br>5. tool output: *greps* *reads*<br>6. assistant: llm gets output for observation<br>7. user: ...<br>8. assistant: ...
My expectation was to hit cache at point 4 for this session - correct, since points 0-3 repeat. But I missed the bigger picture: cache hits can start at point 0 across different users. Your system prompt can be shared across all conversations from your API key org.
My mental model was wrong. I was thinking of inference as a synchronous engine - a single blocking process for one user, like hosting a model locally. First prompt → model does prefill → generates KV cache → responds. Second prompt → hit the cache → fast response.
But this is not how models are deployed at scale by providers like OpenAI and Anthropic. They need to handle concurrent user requests. They do so via async distributed (multi-GPU, multi-node) systems. When the word async comes up, you should get an image of schedulers and message queues in your mind.
These engines incorporate several techniques that optimise LLM inference like KV-cache reuse, continuous batching, chunked prefill, speculative decoding, and more. KV-cache re-use enables prompt caching.
To understand how prompt caching works, we will also need to look at basics of inference engine like vLLM and subsequently how kv-cache re-use is implemented.
Why This Post Exists<br>I could find amazing tips for prompt caching but was unable to find a comprehensive resource on how prompt caching works under the hood. So here I am load-bearing the responsibility and suffering to write the post. Following "Be the change you want to see in the world" etc. When somebody searches "how does prompt caching work really", my hope is this post pops-up and gives<br>them a good idea of how prompt caching works with the bonus of learning how inference looks like at scale.
I spent a lot of time wrapping my head around vLLM engine and inference techniques in the last few days to write this post. This was me a few days back.
Literally me
For a long time I thought prompt caching works due to kv-caching which was partially true. However, it works due to actually<br>re-using the kv-cache via different techniques like paged-attention and radix-attention. In this post, I focus on paged-attention.<br>For that purpose, we would require to look at how vLLM engine works. The aim of this post is to "grok prompt caching" so I will focus on parts of vLLM engine that are super-relevant for paged-attention and prefix-caching.
Before we get to the internals, let me start with tips to hit prompt cache more consistently. These are actually what got me curious enough to figure out how everything works under the hood.
Tips to hit prompt cache more consistently<br>Prompt caching is when LLM providers reuse previously computed key-value tensors for identical prompt prefixes, skipping redundant computation. When you hit the cache, you pay less and get faster responses.
Prompt caching basics and why even worry about it<br>If you use Codex/Claude Code/Cursor and check the API usage, you will notice a lot of the tokens are "cached". Luckily code is structured and multiple queries can attend to same context/prefixes to answer queries so lots of cache hits. This is what keeps the bills in control.
Code generation agents are a good example where the...