Blocket League | World Model Lab<br>J-Lens for video models:<br>uncovering steerable physical dynamics.<br>by Jay Hack·View the code ↗<br>How do video models store information about the physical world?

This lab demonstrates that video transformers can build a compact and interpretable internal models of physics purely from raw video.

We show latent states in video transformers encode position, velocity, collisions and more in geometric representations in their activations. What's more, these representations are "causal" - you can write to them and, in doing so, shape the models' output to form a video game.

This is essentially an application of Anthropic's Jacobian lens to illustrate ideas developed in Interpreting Physics in Video World Models, also inspired by MIRA and Genie 3.

The punchline: steer a video model's hallucinations.

Put the puck in the goal!

The game below is the real-time output of a 3.67M-parameter video transformer running in your browser. Arrow keys write to the model's activations and "steer" the model's hallucinations, using Anthropic's Jacobian lens technique.

Load in-browser model (14.2 MB)

↑←→↓<br>Reset<br>Real-time sampling from a video transformer with steering.

This is not perfect but clearly has grokked the basic physics of the game, including collisions, wall bounces, and scoring. Notably, the model has never explicitly observed input controls - this is distinct from Genie 3 and MIRA, which are explicitly trained with video game controls in mind.

Training a toy video prediction model

To study the learned representation of video models, we can train a tiny video transformer on a toy physics simulation. For this exercise, we use a 3.67M parameter pixel transformer, which directly operates on pixels and predicts the next frame, similar to an LLM predicting the next word.

t−2<br>t−1

Transformer 5 layers

A 5-layer pixel transformer on raw frame data of physics rollouts. View the code on GitHub ↗We train it on Blocket League , a toy physics simulation with collision mechanics between and goal scoring. We sample rollouts and train with a next-frame-prediction objective.

00 goals<br>Reset world

Blocket League

For about $10 of compute, the model cleanly learns the transition function and can “hallucinate” valid game rollouts by recursively sampling frames. The learned dynamics are pretty spot on.

Loading hallucinations…

Generalizing to unseen scenarios

Is this model merely memorizing the training set, or is it able to learn some higher-level representation that generalizes beyond seen examples?

If you retrain this model with certain classes of samples entirely removed - for example, collisions in the top-right quadrant or an entire wedge of movement directions - it turns out the model can still accurately predict what will happen. This is fairly significant and implies the model has learned some higher-level representation that captures entities and their relationships, as opposed to merely interpolating between previously-seen samples.

Loading hallucinations…

Collision physics transfers into the missing quadrant. Against a matched control trained everywhere, the penalty in the unseen quadrant is only 0.8%.

Decoding the learned representation

How accessible is this abstract learned representation?

Some physical properties are easily accessible within the transformer's hidden state. A simple linear regression from block 5 activations, for example, can neatly predict X/Y coordinates of both the orange and the blue ball. See the cross-hairs below for decoded X/Y coordinates, overlaid on next frame prediction samples

Similarly, we can see this same layer will "anticipate" a collision up to eight frames before contact.

Loading coordinate readouts…<br>The crosshairs follow the model's own hallucinations. The same fixed-token decoder keeps tracking both objects even when the model is feeding its own predicted pixels back as input.<br>See the full layer-by-layer and random-weight controls in Appendix D ↓

Motion direction forms a ring in activation space

Motion direction itself has a ring-like topology in the transformer's block-5 activation space.

Average the block-5 activations by motion direction and project them with PCA: they wrap into a circular manifold, with nearby directions in the world remaining neighbors inside the model.

The geometry mirrors what it represents: rotate the physical direction smoothly, and the hidden state moves smoothly around its internal ring.

Principal component projection of direction-conditioned hidden activationsTwenty-four mean block-five MLP activation vectors projected from 768 dimensions onto their first two principal components form a single closed ring. Point color represents observed physical motion direction.PC1PC2187°278°8°98°0°360°observed motion direction

Finding steerable directions with the Jacobian lens.

In NN activations space, readable is not necessarily writable. "Writing" to the x/y activation directions from above does not properly steer...