Doom's renderer, compiled into transformer weights – no training anywhere

Doom, compiled into a transformer — Out of Distribution

Posts · Doom Doom, compiled into a transformer Rob Porter · August 13, 2026 · Code ↗ · Share ↗

Transformers are black boxes … mostly. We normally train them to predict the next token and call the result an LLM. But for years, I’ve wondered what else you could put inside one by constructing its weights directly.

This led me down the road of creating torchwright, a compiler whose output format is a transformer. Feed it a computation graph — a definition of how to get from input tokens to output tokens — and it emits transformer weights. Run inference on the resulting model, and the graph executes. I started by making calculators — simple transformers that evaluate expressions like “12*34”. Once those worked as vanilla Hugging Face checkpoints, I wondered how far I could go. The natural question any nerd would ask: Can it run Doom?

The answer? Yes.

I ported Doom’s renderer into an ordinary LLM architecture. Hugging Face loads the checkpoint with no custom code. Instead of training it to imitate Doom frames, I translated Doom’s original rendering algorithm into that architecture as faithfully as I could. Given a prompt containing the level data, the player’s position, and the viewing direction, the transformer generates the frame Doom would have drawn. My compiler constructed every weight directly from the resulting computation graph. There was no training anywhere.

PROMPT · GAME STATEPROMPT viewx 1056 viewy -3616 viewz 41 viewangle 90 node 0 node.x 1384 node.y -2592 node.dx -40 node.dy -288 node.child1 65 node.child0 93 node.bbox1.top -2592 node.bbox1.bottom -2880 node.bbox1.left 1344 node.bbox1.right 1384 node.bbox0.top -2592

TRANSFORMER MULTI-HEAD ATTENTION FEED-FORWARD

MULTI-HEAD ATTENTION FEED-FORWARD

38 LAYERS

OUTPUT · TOKENS REPLAYED INTO A FRAMEOUTPUT

What exactly did I build?

Before I started, I had to decide what would count as running Doom. I wanted an honest win: the checkpoint itself had to execute substantially the same rendering algorithm as Doom. The map and player state had to enter through the prompt, and the program outside the weights could only convert the model’s output mechanically into pixels.

The result is a stock, decoder-only checkpoint that Hugging Face loads with no custom model code (it uses Phi3ForCausalLM). Generation produces a stream of drawing operations and intermediate values. The host ignores most of them and acts on five drawing commands:

setCursorX(x) and setCursorY(y) move the cursor.

setCursorDirectionX and setCursorDirectionY choose whether it advances horizontally or vertically after a draw.

pixel(color, w) paints a run of w pixels at the cursor using Doom’s palette.

The checkpoint is not tied to a particular frame, player position, or even E1M1. The prompt contains the map geometry, BSP tree, sector heights, texture references, light levels, player position, and viewing direction. Those can all change without recompiling. What is fixed at compile time is the 320×200 output and the texture library (nine wall textures and six floor and ceiling textures chosen for E1M1’s opening scene). The same checkpoint can render any world scene built from those textures; adding another texture requires recompiling.

This is Doom’s renderer, not the complete game. It uses Doom’s low-detail mode, rendering the 3D view in 160 two-pixel-wide columns. Sprites are not implemented, and the weapon and status bar are fixed to Doom’s pistol-start state. For the frame above, 97% of the 64,000 output pixels exactly match the reference renderer.

Below is the model rendering the opening frame of E1M1. On the left is the sequence produced by the transformer. On the right, the host mechanically applies the drawing commands as they are generated.

Outside the weights

The program outside the weights is small. It remembers a cursor, looks up one of Doom’s 256 RGB colors, and paints the horizontal runs requested by the model. The complete 43-line program is below if you want to check that boundary for yourself.

Complete host program 43 lines · Python python · minimal program Copy

import json

from pathlib import Path

from huggingface_hub import hf_hub_download

from PIL import Image

from transformers import pipeline

MODEL = "physicsrob/torchwright-doom-e1m1"

SCREEN = (320, 200)

11 prompt = Path(hf_hub_download(MODEL, "examples/e1m1_prompt.txt")).read_text()

12 colors = json.loads(

13 Path(hf_hub_download(MODEL, "doom_palette.json")).read_text()

14 )["colors"]

16 image = Image.new("RGB", SCREEN)

17 x = y = 0

18 advance_x = False

20 generate = pipeline(

21 "text-generation", model=MODEL, device_map="auto", trust_remote_code=False

23 output = generate(prompt, return_full_text=False)[0]["generated_text"]

25 for token in output.split():

26 command, _, arguments = token.rstrip(")").partition("(")

27 if command ==...

Doom's renderer, compiled into transformer weights – no training anywhere

Related Articles

(no title)

Scientists reverse brain aging, with a nasal spray

NHS apologises and admits Palantir have access to identifiable patient data

AI has torched the market for junior programmers

Al Vigier: Canada's AI strategy shouldn't include secret Palantir bills