World Models 🌍 Explained

TL;DR (for devs eyeing generative 3D/video) - World models skip text and learn the world from raw video, turning a prompt into explorable 3D that's stunning for games and film.<br>What is it? 💡<br>A normal video model just gives you a fixed clip. A world model builds each frame live from your last move plus the frames before it - turn left and it paints what's now on your left, in real time.<br>Usually there's no physics engine and no level designer: the model learns how the world moves by watching a very large amount of video, so its "physics" is a good guess, not a hard rule. Having no real engine is exactly what makes world models renderers that are very visually appealing and hence appear frequently on your social media feeds.<br>A note: some pipelines now bolt a real engine to their models, pushing the model to be more of a "true simulator" and causing physical accuracy to improve dramatically.<br>The frame-by-frame generation cuts both ways: the world is large and expansive, but the memory window is short. Genie 3 can do "promptable events" (like summoning rain mid-walk) because the next frame is malleable. For the same reason, buildings can rearrange themselves when you look away and glance back.<br>World Labs' Marble adds a usable output layer to the renderer, exporting Gaussian splats and a collision mesh that drops into a game or VFX pipeline. Investors already have some serious skin in the game: World Labs has raised ~$1.23B, with Decart AI (of OASIS fame) raising $300M at a ~$4B valuation.<br>I made this: an FPS-style arena for shooting computer bugs, from one text prompt with the Marble API in ~5 minutes.<br>Thoughts 🧠<br>✅ Why - Infinite, promptable worlds. Build a playable level, a film preview, or a simulation from one sentence, no 3D artist required. And you can manufacture the edge cases real footage never captures.<br>🙅 Why not - Physics is hallucinated - people walk backward, and buildings that look flawless from the air fall apart up close. Because perception is the world here, errors cascade frame to frame with nothing to correct them. Coherence lasts minutes, not hours - no persistence yet, and each generation renders a fresh world (i.e. there is not perfect reproducibility). And it's still too expensive for many use cases.<br>⚔️ Players - Genie 3 (DeepMind, now public via Project Genie), Marble from World Labs, Decart (Oasis 3, photorealistic driving over an API at $0.02/sec), and Odyssey (interactive video you steer, a fresh frame every 50ms, and freshly flush with a $310M round). And a different bet: General Intuition trains agents inside game-footage world models (gaming, plus search-and-rescue drones). And the open one: ABot-World (Apache-2.0, 720p on one desktop GPU, playable in-browser).<br>🔮 Prediction - 🤠 Gaming's the obvious lane - personalized single-player and multiplayer genres we've never seen before. The bigger bet: rehearse reality where the real thing is slow, dangerous, or costly - robotics, autonomous vehicles, flight and surgery sims, factory safety. You can also mass-generate adversarial worlds - break your system in sim before reality does. Weird one I like: world models as a codec - send world-state and inputs instead of pixels, compressing bandwidth for cloud-gaming and telepresence. Persistence is the current wall - coherence in minutes, not hours - but most labs are hammering on it, so my bet is it will fall down quickly. Traditional engines don't die - their physics and infra still matter - but they'll have to adapt.<br>Search interest in "world models", past 5 years. Source: Google Trends.<br>Further reading 📚<br>A Functional Taxonomy of World Models - Fei-Fei Li on what the term even means (renderer vs simulator vs planner). Best explainer on the whole topic.<br>Why you can't trust a hallucinated world - the skeptic case, sharply argued.<br>Decart's Oasis 3, with some caveats - what API-first world models look like in practice.<br>World models, from scratch - video walkthrough of the original idea (vision model + memory + controller).<br>Awesome World Models - a 3k-star curated hub, renderers to robots, for the full rabbit hole.<br>Thanks to @TomGranot, who edits every issue and remains the most physically-faithful simulator I know.

Agam More

Don't miss the next dev trend 👻

4,100+ devs have already subscribed.

Enter your email

Subscribe

You might also like...

Powered by Ghost