bageldotcom/worlddit · Hugging Face
Log In<br>Sign Up
WorldDiT: A Unified Diffusion Backbone for<br>World and Action Modeling
WorldDiT couples continuous action generation with auxiliary future normalized<br>RGB patch prediction in one diffusion transformer. The architecture is designed<br>as a general backbone for world and action modeling, while the current release<br>evaluates it on LIBERO and provides four checkpoints, a self contained<br>inference runtime, and an evaluator.
See WorldDiT act
The four clips below show successful rollouts from the released checkpoints.<br>Each clip covers a different LIBERO suite and camera view.
LIBERO Spatial
Task 5, front view.
LIBERO Object
Task 8, agent view.
LIBERO Goal
Task 10, side view.
LIBERO Long
Task 6, front view.
Spatial MP4<br>· Object MP4<br>· Goal MP4<br>· Long MP4
Release snapshot
Reported LIBERO result<br>Released model
94.9 percent mean success<br>1,898 of 2,000 successful episodes<br>399.084 million total parameters<br>135.107 million trainable parameters
98.0 percent Spatial<br>97.0 percent Object<br>Three observation steps<br>Seven predicted actions
92.8 percent Goal<br>91.8 percent Long<br>Three actions executed before replanning<br>Seven action dimensions
Checkpoints<br>Runtime<br>Encoders and environment
Spatial<br>Object<br>Goal<br>Long<br>inference.py<br>eval.py<br>config.json<br>MAE ViT B<br>OpenAI CLIP ViT B 32<br>SafeTensors and pinned requirements
The repository is self contained for WorldDiT inference. LIBERO provides the<br>benchmark environments, assets, task definitions, and initial states.
The released runtime and checkpoints were revalidated from a clean installation<br>on eight RTX Pro 6000 Blackwell GPUs. The reported aggregate covers five<br>hundred simulator episodes per suite. Three hundred episodes per suite informed<br>staged checkpoint selection, while two hundred episodes per suite were disjoint<br>from selection.
Parameter count and reported success
Among methods with complete four suite averages, WorldDiT lies on the reported<br>Pareto frontier for total model parameters and mean LIBERO success.
Reported LIBERO success against total model parameters for 24 methods. The line connects methods on the Pareto frontier with complete four suite averages. Because the methods follow different published evaluation protocols, the figure summarizes published results rather than a direct comparison under one common evaluation protocol.
Run a smoke test
Download the repository and create a clean Python 3.12 environment.
hf download bageldotcom/worlddit --local-dir worlddit<br>cd worlddit
python3.12 -m venv venv<br>source venv/bin/activate<br>python -m pip install -r requirements.txt<br>python -m pip install --no-deps robosuite==1.4.1
LIBERO supplies the benchmark definitions, assets, and initial states. Keep the<br>checkout at ~/LIBERO, which is the evaluator's default.
git clone https://github.com/Lifelong-Robot-Learning/LIBERO.git ~/LIBERO
The released evaluation was validated with LIBERO commit<br>8f1084e3132a39270c3a13ebe37270a43ece2a01.
python eval.py \<br>--suite libero_spatial \<br>--gpus 1 \<br>--tasks 1 \<br>--episodes 1 \<br>--max-steps 20 \<br>--output-dir results/smoke
A successful smoke test confirms that the environment, checkpoint, visual<br>encoders, simulator, and rendering path load together. Full benchmark reporting<br>uses complete suite evaluations.
How WorldDiT works
Each of three recent observation steps contributes primary and wrist images<br>together with robot state, while one language instruction conditions the<br>sequence. During training, one diffusion transformer backbone learns a seven<br>step action chunk together with an auxiliary future normalized RGB patch<br>target. At deployment, the encoded history conditions the action velocity field<br>directly. RGB patch token construction and RGB prediction head evaluation<br>remain outside the inference graph, concentrating computation on action<br>generation. The controller executes the first three predicted actions, observes<br>again, and replans.
WorldDiT inference pipeline. The encoded observation history conditions action generation through twenty flow steps. The controller executes the first three actions from each seven action chunk, then updates the window and replans.
Training<br>Deployment
Action and future normalized RGB patch targets are learned by one backbone<br>Encoded history conditions the action velocity field
Seven action steps are supervised<br>Seven actions are predicted
Future normalized RGB patch supervision is present<br>RGB patch tokens and the RGB prediction head remain outside the inference graph
The complete training objective is active<br>Three actions execute before replanning
Reference
Full evaluation commands
One GPU
python eval.py \<br>--suite libero_spatial \<br>--gpus 1 \<br>--output-dir results/libero_spatial
Multiple GPUs
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python eval.py \<br>--suite libero_spatial \<br>--gpus 8 \<br>--output-dir results/libero_spatial_8gpu
Each GPU receives an independent progress bar. After all workers finish, rank 0<br>prints per task and overall success rates...