Simulating everything, sort of: The promise and limits of world models - Ars Technica

Skip to content

AI

Biz & IT

Cars

Culture

Gaming

Health

Policy

Science

Security

Space

Tech

Forum

Subscribe

Story text

Size

Small<br>Standard<br>Large

Width

Standard<br>Wide

Links

Standard<br>Orange

* Subscribers only

Learn more

Pin to story

Theme

Search

Sign In

Sign in dialog...

Text<br>settings

Story text

Size

Small<br>Standard<br>Large

Width

Standard<br>Wide

Links

Standard<br>Orange

* Subscribers only

Learn more

Minimize to nav

Over the past few years, many of us have gotten a crash course in what we now call artificial intelligence—but really, it has mostly been a crash course in large language models. Increasingly, however, LLMs are no longer the only category of AI drawing high expectations, massive funding rounds, and significant research and product development.

Over the past year, we’ve seen a plethora of new announcements in a category labeled “world models,” and you’ll likely see more movement there in the coming months and years.

Instead of or in addition to working with language, world models aim to lay the groundwork for AI systems that are capable of simulating the physical world, or at least a useful approximation of it.

To examine what’s different and important about this idea, Ars spoke with three expert practitioners working on world models and related technologies: Vincent Sitzmann from MIT, Anastasis Germanidis from Runway, and Ben Mildenhall from World Labs.

From these conversations, we learned that while LLMs-as-a-product started with an interface (chat) and then sought a use case, the big players in world models right now are arguably working in the other direction: They’re starting with specific use cases and applications in robotics, research, and asset generation, but it’s unclear exactly how the interfaces, systems, and tools will ultimately look.

Table of Contents

Jump to section

The off-ramp from LLM disillusionment

As you’ll soon see, there are many parallels between LLMs and world models in terms of architecture and how people expect them to improve over time. For some, though, they’re seen as a potential answer to the limitations of LLMs, even though work on them predates that contemporary narrative.

“The idea that you’re going to extend the capabilities of LLMs to the point that they’re going to have human-level intelligence is complete nonsense,” former Meta chief AI scientist Yann LeCun told Wired earlier this year. LeCun has made waves with an opinion that some working in AI and LLMs see as contrarian, but he’s actually speaking for a sizable segment of the field.

See also Fei-Fei Li, the computer vision pioneer who co-founded World Labs, one of the new companies working on world models. In a Substack post late last year, she wrote:

Today, leading AI technology such as large language models (LLMs) have begun to transform how we access and work with abstract knowledge. Yet they remain wordsmiths in the dark; eloquent but inexperienced, knowledgeable but ungrounded. Spatial intelligence will transform how we create and interact with real and virtual worlds—revolutionizing storytelling, creativity, robotics, scientific discovery, and beyond. This is AI’s next frontier.

LeCun and Li’s ventures are built on these ideas, so it’s not surprising they’d say these things. But you’ll also see similar sentiments from some prominent figures still working primarily with LLMs.

“I think we’re in an LLM bubble, and I think the LLM bubble might be bursting next year,” said Clem Delangue, the CEO of Hugging Face—a platform that hosts repositories of LLMs of all stripes.

“But ‘LLM’ is just a subset of AI when it comes to applying AI to biology, chemistry, image, audio, [and] video,” Delangue added, speaking at a conference. “I think we’re at the beginning of it, and we’ll see much more in the next few years.”

A financial flurry

Over just the past few months, world models have advanced from a research topic (which they still are, of course) to the basis for new commercial projects and huge funding rounds. A few key examples:

In August, Google DeepMind unveiled Genie 3, a model that builds real-time interactivity on top of the foundation of a video generation model.

In November, World Labs introduced Marble, a model and toolset that allows users to generate immersive environments that can be exported as 3D assets, based on input in the form of text, images, video, or other assets.

Just a month later, video generation and filmmaking AI company Runway entered the fray with its announcement of GWM-1, a trio of specialized world models built on Runway’s past work with video models.

Even more recently, Yann LeCun started Advanced Machine Intelligence (AMI), a company betting the farm on the notion that the real future of AI systems is in models that interact with the physical world (or at least simulate it), not just language.

These and similar efforts have received substantial...