How a Transformer Plays Tic-Tac-Toe
In 2017, the paper Attention is All You Need introduced the Transformer architecture. It is now the foundation of modern large language models. Text prediction requires complex models, and what happens inside them is difficult to see. This article uses fading Tic-Tac-Toe instead. The game is simple enough that even a basic model works well, and every step is easy to follow.
Play<br>1-head2-headno-posno-causalno-MLPno-res
012345678
ResetWait…
Temperature🤔<br>0.42
embedding
pos.encoding
attention
MLP
Unembedding
The goal is to get three in a row before your moves fade. Only the last 6 moves stay on the board, older ones vanish. Moves about to disappear turn light gray, the latest played move is green.<br>Each square is numbered 0 to 8, so a game can be encoded as a sequence of moves:X5 O6 X4 O3 X0 O8 X2 O1 X6 O4 X7 …Because old moves disappear, the game can keep going for as long as it takes for someone to win. It also makes move order matter. In standard Tic-Tac-Toe only position counts. Here, when a move was played matters too.<br>The Transformer receives this sequence and predicts the next move. Six models are available. Two are complete architectures, four are ablations with one component removed:<br>1-head — single-head attention, 32-dimensional.<br>2-head — two-head attention, 16 dimensions each.<br>no-pos — positional encoding removed. The model sees tokens but not their order.<br>no-causal — causal mask removed. The model can attend to future positions during training.<br>no-MLP — MLP block removed. Attention is the only transformation.<br>no-res — residual stream removed. Each block overwrites rather than accumulates.
Architecture<br>The Transformer consists of six main blocks: tokenizer, embedding, positional encoding, attention, MLP, and unembedding.<br>It processes a sequence of moves in parallel — one vector per token. Each vector flows through the blocks via the residual stream, which carries information forward so that every block adds to the representation instead of overwriting it. At the end, each vector produces a prediction. Only the last one is used to choose the next move (the others are used for training). It gets appended to the sequence, and the process repeats.<br>Attention is the only block where tokens interact with each other. In every other block they are processed in total isolation.<br>Blocks are duplicated to show they operate independently on each token. Only attention uses all tokens at once.<br>Tokenizer — Converts moves into integers.<br>Embedding — Maps each integer to a 32-dimensional vector.<br>Positional encoding — Marks when each move was played in the sequence.<br>Attention — The communication layer where tokens read information from each other.<br>MLP — A reasoning step that refines the representation of each move.<br>Unembedding — Turns each vector into scores over possible next moves.
Tokenizer<br>The model works with vectors stored in a matrix, so it needs a numerical index to fetch them. The tokenizer maps each move to that integer index.<br>The 20-token vocabulary includes 9 squares for X and 9 squares for O plus special tokens START and PAD<br>moves 0
vocabulary 20
tokens 0
X0 → 0<br>X1 → 1<br>X2 → 2<br>X3 → 3<br>X4 → 4<br>X5 → 5<br>X6 → 6<br>X7 → 7<br>X8 → 8<br>O0 → 9<br>O1 → 10<br>O2 → 11<br>O3 → 12<br>O4 → 13<br>O5 → 14<br>O6 → 15<br>O7 → 16<br>O8 → 17<br>START → 18<br>PAD → 19
Play a few moves in the game to visualize.
Embedding<br>The embedding layer maps each token to a point in a 32-dimensional vector space the model can do math with.<br>tokens 0
embedding table 20×32
embeddings 0×32
10
11
12
13
14
15
16
17
18
19
Play a few moves in the game to visualize.
These vectors are learned. They shift during training until the model finds a useful arrangement.
Positional Encoding<br>The math in the attention block is order-agnostic. Without help, the model cannot distinguish between sequences like X1 O2 and O2 X1.<br>Positional encoding fixes this by adding a unique vector to each move based on its position in the sequence. This "stamps" every token with its timing to keep track of move order.<br>embeddings 0×32
positions 16×32
residual stream 0×32
Play a few moves in the game to visualize.
The model uses learned positional embeddings (as in GPT-2). These parameters shift during training to represent the game's timeline.
Attention<br>Attention is the communication layer. Each move looks back at the sequence to gather context about the game state. The entire exchange is described by a single formula:<br>z=softmax(QK⊤+Md)Vz = \text{softmax}\left(\frac{QK^\top + M}{\sqrt{d}}\right)Vz=softmax(dQK⊤+M)V<br>This formula says: compare every move to every other move (QKᵀ), restrict to the past (M), turn scores into weights (softmax), and use them to mix information (V).<br>Before computing attention, the vectors are normalized so all values are on a comparable scale for stable training.<br>residual stream 0×32
normalized 0×32
Play a few moves in the game to visualize.
Note: Layer normalization is not necessary for this small model, but it is...