Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation<br>🔭">

Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

Alexi Gladstone1,<br>Heng Ji1,<br>Yilun Du2

1UIUC 2Harvard

Paper

arXiv

Code

Blog

HF Paper

Tweet<br>SOON

alphaXiv

TL;DR

We introduce Explorative Modeling , a new paradigm for generative modeling that acts as a third pretraining axis when added to existing generative models, and also enables end-to-end generation . Increasing exploration monotonically improves existing models across images, video, and language, and the gains grow with scale (7%→36% with data, 13%→23% with parameters). Concretely, Explorative Models (XMs) reach 6.2× sample efficiency, 4.1× FLOP efficiency, and 47% better parameter efficiency. Exploration also enables scaling generalization, and scaling how end-to-end existing models are. As end-to-end generative models, XMs match diffusion on control tasks with up to 256× less inference compute.

Overview Video

Your browser does not support the video tag. Download the video.

Abstract

The Idea: Factor Training, Not Generation

A generative model can break two things into pieces: how it generates, or how it trains. Today's models break up generation into hundreds of small steps, which works but prevents end-to-end generation. Explorative Models (XMs) break up training instead, through exploration. Because of this, for existing generative models, XMs unlock a third pretraining axis beyond parameters and data. XMs also enable end-to-end generative modeling, where sampling during training and inference are identical.

Two axes of generative modeling.

More exploration → less blur

Ground Truth

XM-1

XM-2

XM-5

XM-50

Ground Truth

XM-1

XM-5

XM-20

XM-50

Ground Truth

XM-1

XM-2

XM-4

XM-8

Increasing exploration (K) lets a model capture more modes instead of averaging them.

Exploration as a Third Pretraining Axis

We add exploration to RAE, a near-SOTA image-generation recipe. Without any hyperparameter tuning, simply adding XM gives 6.2× better data efficiency and 4.1× better FLOP efficiency. The compute-optimal amount of exploration also keeps growing the longer you train.

Adding exploration to RAE reaches the no-exploration baseline's best quality with 6.2× less data (left) and 4.1× fewer FLOPs (right); quality is FDr⁶ (lower is better).

The same holds for model size: a Large model that explores 5 modes outscales an XLarge model with 47% more parameters and no exploration.

A Large model exploring 5 modes scales better than an XLarge model with 47% more parameters and no exploration.

With the model fixed, increasing exploration alone monotonically improves performance across images (FID), video (FVD), and language (a masked-diffusion LM). Some models gain over 20%, and the gains have not stopped at the highest exploration levels we test.

As modes explored increases, both FID (left) and FVD (right) improve for Explorative Diffusion (XDiffusion) and Explorative Jumpy (XJumpy); the more end-to-end model (XJumpy) benefits more.

It works for text too: adding exploration to a masked-diffusion language model (XMDLM, 8 modes) lowers perplexity at every diversity level, for both fast 8-step (left) and slower 256-step (right) generation.

A generative model's performance is limited by three capacities: parameters restrict what it can represent, data restricts what it can learn, and generative expressivity restricts what it can generate. Scaling parameters and data relieves the first two, but generative expressivity is set by the training objective itself, so as models and data grow it increasingly becomes the bottleneck. Exploration raises generative expressivity directly, and its gains grow with scale — rising from 13% to 23% as models scale and from 7% to 36% as data scales. This is what makes exploration a genuine third pretraining axis : just like parameters and data, the compute-optimal amount of exploration grows with scale, so models trained without it fall increasingly short.

The improvement from exploring 5 modes grows with model size (left, 13%→23%) and data (right, 7%→36%), the signature of a real scaling axis.

Because exploration commits to real modes instead of a blurry average, there's less of an artifact to memorize, so models generalize better. On a small video dataset, more exploration overfits less and reaches a better best FVD (30.0 vs 37.5).

Tracking held-out video quality over training: more exploration memorizes the training set less and reaches a better best FVD (30.0 at XM-12 vs 37.5 with none).

Exploration also lifts the strongest recipes. Added to RAE (state-of-the-art for ImageNet 256×256 shortly before this work), it reaches a near-best 1.43 FID without guidance and converges ~300× faster than the standard SiT recipe.

Method (ImageNet 256×256, no guidance)<br>gFID ↓<br>FDr6 ↓<br>IS ↑

REPA-E (VAE latent diffusion)<br>1.70<br>217.3

DiTDH-XL (RAE baseline)<br>1.55<br>4.42<br>237.3

XDiTDH-XL, XM-2...