kira-project-site/posts/2026-07-06-prefill-tax.md at main · thekiraproject/kira-project-site · GitHub
//blob/show" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
//blob/show;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
thekiraproject
kira-project-site
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
FilesExpand file tree
main
/2026-07-06-prefill-tax.md
Copy path
Blame<br>More file actions
Blame<br>More file actions
Latest commit
History<br>History<br>History
101 lines (58 loc) · 10.9 KB
main
/2026-07-06-prefill-tax.md
Copy path
Top
File metadata and controls<br>Preview
Code
Blame
101 lines (58 loc) · 10.9 KB
Raw<br>Copy raw file<br>Download raw file
OutlineEdit and raw actions
Your local AI is re-reading its own prompt
Measuring — and eliminating — the prefill tax in a local AI pipeline. The Kira Project, July 2026. Patent pending (U.S. provisional 64/105,533).
Everyone benchmarks local LLMs the same way: decode tokens per second. Nobody watches the other number. On our system, the other number was nearly half of all model time — and it was being spent re-reading text the model had already read, thousands of times a day, invisibly.
Quick definitions, then the story. An LLM answers in two phases: prefill reads — the entire prompt is processed token-by-token into the model's working state before the first output token can exist (that's the pause before the answer starts) — and decode writes, one token at a time against that state. "Tokens per second" measures decode only. Prefill cost scales with prompt length, and in a multi-stage pipeline the prompt is mostly boilerplate the model has read ten thousand times before. Picture a lawyer who re-reads the entire case file before answering each deposition question.
This is the story of finding that tax, measuring exactly what it cost, and removing it with a discipline we think matters more than the cache itself. Everything below is measured on real hardware, on a production system, with the receipts in a public evidence ledger.
The setup
Kira is a local compound-AI assistant: one 9-billion-parameter model (8-bit quantized, hybrid attention) serving every pipeline role — router, specialist, synthesizer, agentic planner — on an Apple M4 Pro with 24 GB of unified memory, through Apple's MLX framework. No cloud calls, ever. A query flows through a classifier, a reasoning loop with ~50 live tools, and a synthesizer; each stage is the same model wearing a different prompt.
That "different prompt" detail is where the story hides.
The regression nobody reopened
Every month of this project's history — five of them, at the time of writing — is recorded in architecture decision records, and one of them contains a finding that quietly inverted. Early on, we benchmarked KV-cache prefix reuse and concluded we had it for free: our then-runtime (Ollama, on llama.cpp) automatically reused the key/value state for any request sharing a token prefix with the previous one. Static stage prompts + pinned models = every call after the first skipped its shared prefix. The same research noted that MLX had no built-in cross-request prefix caching. Both findings were correct that day.
Then we migrated runtimes — Ollama out, in-process MLX in, for independently excellent reasons (93 s vs 180 s per query, no phantom model loads). The migration silently deleted the automatic prefix reuse , and nobody reopened the old finding. Worse: the new client reported prompt_eval_count: 0 on every call. The cost wasn't just unpaid attention — it was literally invisible to our own instrumentation.
Lesson one: when you change runtimes, re-audit every "we get X for free." The free things don't announce their departure.
Instrument first
Before building anything, we surfaced the inference engine's own per-call statistics — prompt token counts and prefill durations — into our telemetry. No mechanism, no cache, just eyes. What they saw, on the live broker:
Pipeline tier<br>LLM calls<br>Prompt tokens<br>Prefill time<br>Decode time<br>Prefill share of model time
standard<br>8,880<br>20.0 s<br>22.4 s<br>44%
agentic<br>5,749<br>13.1 s<br>14.4 s<br>47%
research<br>12<br>21,989<br>49.7 s<br>231.5 s<br>17%
And a critical companion fact: after the...