MediumWhy prompt injection works: a Transformer-level view | by Kirill | Jun, 2026 | MediumSitemapOpen in appSign up<br>Sign in

Medium Logo

Get app<br>Write

Search

Sign up<br>Sign in

AI

Cybersecurity

Vulnerability

LLM

Machine Learning

Why prompt injection works: a Transformer-level view

Kirill

7 min read·<br>Jun 23, 2026

Listen

Share

Press enter or click to view image in full size

Preamble<br>A few weeks ago, I wrote about the anatomy of prompt injection. The thesis was simple: data and instructions share the same context window , and that’s the root of the problem.<br>A reader could reasonably ask: “Why? It’s just text. Surely we can tell the model which part is instructions and which is data?” That’s a good question. The honest answer is no, not at the level where it would matter. And to see why, you have to look at the model's architecture.<br>This is a short follow-up. We go one layer deeper. Not a Transformer tutorial — if you want a gentle intro to Transformers, Jay Alammar’s Illustrated Transformer is the canonical one. Here, just enough internals to explain why no amount of clever prompting really fixes prompt injection.<br>The model sees one stream of tokens<br>The first thing to know: when you “send a prompt” to an LLM, it doesn’t see your message as a structured object. It sees a single sequence of tokens — numbers in a list.<br>[14897, 395, 6085, 495, 4109, 1926, 0, ...]That’s it. No `role: “system”` field. No `is_user_input: true` flag. No type system. Just one flat stream.<br>Press enter or click to view image in full size

OpenAI-style chat formats with `{role: “system”, content: “…”}` and `{role: “user”, content: “…”}` look structured on the API side — but before they reach the model, they get serialized into the same flat token stream with special separator tokens between them. The model learns during training that certain separator tokens often precede an “instruction” and others often precede “user content.” That’s a statistical pattern in the training data , not an architectural boundary.<br>Which is the first reason prompt injection is hard to fix: the model doesn’t have a built-in concept of “trusted” tokens versus “untrusted” tokens . Everything is tokens. The same machinery processes them.<br>A quick primer: how attention works<br>Each token is really a vector of properties. Picture “lemon” as a list:<br>lemon = [Yellow: 0.9, Sour: 0.8, Oval: 0.5, Fruit: 1.0]And “sugar”:<br>sugar = [White: 0.9, Sweet: 1.0, Crystalline: 0.7, Food: 1.0]In reality, the dimensions don’t have nice human names — they’re learned. But the intuition holds: each token lives at some point in property space.<br>In an attention layer, every token’s vector is projected through three weight matrices (W^Q, W^K, W^V) into three different views of itself. Each projection lives in the same property space — it just zeroes out some dimensions and amplifies others, exposing one specific facet of the original token.<br>Q (Query) — what this token is looking for in others . ”I’m looking for something sweet.”<br>K (Key) — what this token advertises so others can find it. ”I’m sour and yellow.”<br>V (Value) — what this token actually delivers if found. Where K is the title on the spine of a book, V is the chapter inside.<br>How does the mixing happen? Every token’s Q is compared against every other token’s K — a simple dot product, one number per pair. Higher dot product = better semantic match. Those numbers are normalised into attention weights that sum to 1 — a soft preference distribution across the rest of the context.<br>The token’s new vector is then a weighted sum of the V vectors of everything in the context, using those attention weights.<br>So Lemon’s updated vector is part old-Lemon, plus a big chunk of Sugar’s V (because Lemon’s Q matched Sugar’s K well), plus tiny bits of every other token (very small weights). The mixing happens across the whole context in one step — every token can absorb a little of every other token. After a few layers of this, each token has been shaped by everything it had in common with.<br>That’s the picture. Everything below builds on it.<br>Attention treats everything the same way<br>That same machinery runs over the whole token stream, regardless of role.<br>Here’s the important detail. No rule says “system-prompt tokens can influence user tokens but not the other way around.” Attention is symmetric in the sense that any token can attend to any other token in its context window. If a user-supplied data token has a key that matches the query of an instruction-following token, attention will flow.<br>This is the second reason prompt injection is hard to fix at the model level. The mechanism that lets the model “follow instructions” is the same mechanism that lets an attacker’s tokens reach into the instruction-following part of the computation.<br>Press enter or click to view image in full size

The same Q/K/V matrices process everything<br>When the model processes a token, it transforms its embedding into Q, K, and V using three weight matrices: W^Q, W^K,...