MIRA - Blog post<br>← Back to lobbyMIRA: Multiplayer Interactive World Models with Representation Autoencoders#

DemoTechnical reportCodeDataset<br>by General Intuition & Kyutai<br>in collaboration with Epic Games

Hover/tap on one quarter to zoom.

We train a model to simulate Rocket League, Epic Games' car-football game.<br>It allows four players to play a 2v2 match together, responding in real time to their actions and generating a consistent view of the game at 20 fps, with a resolution of 576p split across the four players.<br>It's a 5B-parameter diffusion transformer paired with a 600M-param video representation codec. We're open-sourcing the dataset as well as training and inference code.

The project is a stepping stone to physical AI, where data is messier and scarcer.<br>Before we can make things work in the controlled video game setting, why would we hope to succeed on real-world data?<br>So this is a demo to show what's possible; the world model tech is not being used to develop Rocket League.

Play<br>It works#

The model manages to simulate many aspects of Rocket League.<br>The cars move, they can kick the ball, you can score goals. The four cars move in sync on the four screens.<br>The model tracks how much boost you have and doesn't let you boost when you're out.<br>Demolitions work.<br>It shows event messages like "Shot on goal" or "Save".

Pressing "boost" only works until you run out (gauge in the bottom right).<br>That sounds trivial when said out loud, but it's a small wonder it works:<br>MIRA has no physics engine, no rendering engine, and no explicit 3D representation at all.<br>It's just videos and actions crammed into a transformer that learns everything purely from data.

Autopilot#

We train the model using action dropout, meaning that during training we hide some of the keys that the players were pressing and just tell the model to predict what would happen without this information.<br>That allows us to have an option to use autopilot mode in the demo.<br>Since the data is gameplay footage of a strong bot (Nexto), the model predicts what the bot would have done.<br>This makes four-player autopilot showdowns quite a sight to behold:

Good luck touching the ball.<br>Stability#

One of the most remarkable things about the model is how stable it is: as far as we can tell, it can run infinitely without diverging.<br>A longstanding challenge in causal video models like this is staying stable over longer rollouts; most models lean on dedicated tricks and multi-stage training to avoid diverging, whereas we get stability with a fairly simple recipe (explained in the Architecture section below). This is likely helped by our relatively narrow visual domain so the approach might not generalize, but it's a strong outcome nonetheless.

The model can keep going forever. In this video, we let four autopilots play against each other without resetting the model after each goal like we do in the demo - we just keep going. Replays are funky but the model does reset to a kickoff each time.<br>Why simulate video games?#

A natural question to ask is: what is the point of simulating video games in the first place?<br>Why play MIRA when you can just play Rocket League?

To which we say: You should play Rocket League!

Our work is not meant to replace video games with AI models.<br>The reason researchers are excited about world models is rather physical AI: think robotics and self-driving.<br>In those domains, it's expensive to collect data and testing the models is dangerous, since you don't want to crash a car or break a robot.<br>If we train a world model that predicts with high accuracy how a car would behave in the real world driven by a model we're testing, we can evaluate it by running it in the world model, which is safer and quicker. If the world model is accurate enough, you could even directly train in the world model.

Simulating video games is therefore a stepping stone to physical AI: it's a setting where data is abundant and it's cleaner than real-world videos since it's generated by a game engine.<br>But learning the mechanics of an AAA video game purely from observing video is still no easy feat, so the hope is that what we learn by studying video game world models will be useful in real-world settings later.

A more direct reason for why video games can be helpful for physical AI is what's known in the business as sim-to-real transfer.<br>The argument goes that video games are already somewhat similar to the real world: Rocket League is basically football with cars,· so maybe a model that understands it would also have an easier time understanding cars and/or football.<br>More specifically, you could pre-train a model on a large volume of video game data and then fine-tune with a much smaller corpus of real-world data, and the model would work significantly better than if trained purely on the real-world data.·<br>We don't yet know how far this sim-to-real transfer can take you when scaled up.

Data#

We train on roughly 10,000 match-hours of recorded 2v2 Rocket League...