Scaling Video Pretraining with Imagination Models — Induction Labs

Internet video contains millions of hours of people using computers, acting in the physical world, interacting with one another, and performing skilled work.

We introduce imagination models, a simple foundation model architecture that unlocks scalable learning from this internet video. These models learn to imagine the future in a learned representation space. During pretraining, imagination models implicitly learn to act despite seeing no action labels. Afterwards, reinforcement learning can be scaled to consistently improve their competence.

We test the imagination model architecture with Photon-1, a sparse 106B-A5B MoE transformer trained on 18 years of computer demonstration video. On internal computer use benchmarks, Photon-1 outperforms a production LLM trained with 30x more FLOPs, and is 3x cheaper to serve . And despite seeing only computer use video, it serves as a basis for broader world understanding; for example, it can learn to play checkers and simulate billiard physics better than LLMs. It also demonstrates desirable human behavior present in pretraining data, such as steering AI tools to complete tasks.

Simulate Computers

Photon-1 can act as a computer world model. After a single seed screenshot, all subsequent states are generated based on the action shown.Model Physics

After finetuning on billiard videos, Photon-1 can simulate complex physical interactions. Billiard ball positions are predicted by the finetuned model.Use ChatGPT Like a Human

After reinforcement learning, Photon-1 learns to use ChatGPT like a human, steering the LLM to get what it wants.Play Checkers

Photon-1 can play checkers after finetuning on 20,000 tournament games, beating LLMs finetuned on the same data.

Using a Computer<br>Game CatalogWebsite CodingYouTube InteractionDocument Formatting<br>Task: Open the Steam page for Counter-Strike 2.

Photon-1 can be finetuned to imagine the next computer state that works towards solving a task, then act on that imagination. The imagination is predicted in representation space, then visualized with an auxiliary finetuned image generation model.

Learning from observation

Humans can learn by watching other people. What we learn from is often not the raw actions a person takes, but higher level concepts of what the person is doing.

Video contains an enormous amount of this observational data, which imagination models can learn from. Our key result is that predicting future states can teach models to complete tasks despite never seeing actions during pretraining, allowing them to learn broadly from video through observation alone.

Photon-1 is our initial test of imagination models, trained on only computer use videos. We use computer environments as a testbed because they are open-ended, cheap to run at scale, and provide access to a large share of humanity's knowledge and work.

Training imagination models

Imagination models predict future frames autoregressively using a next-latent-token-prediction objective. They do not generate pixels during pretraining; everything is modeled in representation space.

Our encoder converts each frame into a set of latent tokens which capture its semantic meaning. The imagination model predicts the next latent token autoregressively . After predicting enough tokens, the model will have predicted the latent for the next video state.<br>Our approach is made possible by our vision encoder, which uses finite scalar quantization (FSQ) to compress each frame into 960 discrete tokens. Each token is an 8‑dimensional vector, where each dimension takes one of five values {−1, −1/2, 0, 1/2, 1}, yielding a vocabulary of 5⁸ possible codes. The encoding totals just 2.2KBlog⁡2(58)×960\log_2(5^8)\times960log2​(58)×960 bits, providing 100× better compression compared to existing OCR and multimodal-model representations , while preserving the text, layout, and state changes needed to understand video.

Our vision encoder achieves over 100x compression in size of state representation, and maintains competitive performance in text and temporal difference tracking.<br>To achieve this rate of compression, Photon-1 uses a differential latent encoder, which encodes video frames as pairs to allow the latents to describe differences between frames.

Our differential latent encoder encodes the differences between frames rather than the contents of the frames themselves.<br>Our pretraining corpus originates from an internal index of 2 billion publicly available videos, filtered down to approximately 2 million computer screen recordings. We performed extensive video and frame-level filtering to remove non-computer-use content, and built an internal keyframe detection model to remove redundant frames. We pretrained Photon-1 from scratch for a single epoch on the resulting 575-million-frame dataset, equivalent to 552 billion tokens or 18 years of video sampled at 1 frame per second.

Pretraining the 106B-A5B MoE...