TokenTown: How an LLM Works, as a City

Passprefill

Layer0 / 6

KV cache0 tokens

Generated0

Details

idle

Press Run to start

The city is idle. Type a prompt below and watch a single token make the round trip.

reading stop: press Space to hold it here

Residual stream idle

12 numbers standing in for the 4,096+ a real model carries. Blue is negative, warm is positive.

Attention softmax over the KV cache

Runs at the Attention Plaza.

Next token logits → softmax

Computed at the Vocabulary Stadium.

Context

Output

nothing yet

Districts click to fly there

What this is

TokenTown is an isometric city where every district is one stage of a transformer language model.<br>A convoy carries a hidden state along the roads: it is cut into tokens at the docks, cast into a<br>vector at the foundry, stamped with its position, then driven around the layer ring (attention, residual,<br>feed-forward, residual) once per layer, before the stadium turns it into a probability distribution and<br>the sampler picks one token. That token drives back up the feedback highway and the whole city runs again.

How much of it is real

Genuinely computed, live, in your browser: the tokenizer split; the embedding lookup;<br>sinusoidal positional encoding; LayerNorm; multi-head scaled dot-product attention with causal masking<br>over a real growing KV cache; the residual adds; a GELU feed-forward; and temperature / top-p sampling.<br>The bars on the truck are the actual vector. The beams over the warehouse are the actual softmax weights.<br>Prefill really does process every prompt token at once while decode really does process only one.

Scaled down: 12 dimensions instead of thousands, 2 attention heads instead of dozens,<br>2–12 layers instead of 80, and a vocabulary of a few hundred words instead of 100k+.

Deliberately faked: the weights are random, and nothing here was trained, so a random-weight<br>model would emit noise. To keep the output readable, the final logits blend the real hidden-state<br>projection with a bigram prior built from a small fixed corpus. The attention scores are also sharpened,<br>and given a small first-token ("sink") and recency bias, so the map looks like the patterns trained models<br>actually produce. Treat the text this city writes as scenery; treat the mechanism as the lesson.

Pacing

The first time the convoy reaches a district it stops long enough to read that<br>district's explanation, between 9 and 26 seconds depending on how much there is<br>to say. A progress bar under the panel text shows how long the stop has left.<br>Once every district has been explained the city runs at a watchable pace instead<br>of a readable one, and the repeated layers fast-forward because they are the same<br>road with different weights. Space holds any stop indefinitely, S<br>steps one stage at a time, and the Speed slider scales everything, including the<br>reading stops, from 0.4× to 8×. Reset (⟲) replays the slow tour from the<br>beginning; Run keeps what you have already read.

Controls

Space play / pause · S advance one stage · R reset and replay the tour · F follow camera · L labels

Drag to pan, scroll to zoom, click any district for its explanation.

The view starts close on the convoy and rides along with it. The ⤢ button<br>on the left (or a double-click on the map) pulls back to the whole city; turning<br>off Follow lets you wander on your own.

Inspired by the idea behind PGSimCity (a city-shaped model of PostgreSQL); all code, art and copy here are original.