Dream-Cubed: Controllable Generative Modeling in Minecraft by Training on Billions of Cubes

This page requires JavaScript to render the full Distill layout.

Dream-Cubed<br>Controllable Generative Modeling in Minecraft by Training on Billions of Cubes

Summary

We introduce Dream-Cubed, a new large-scale dataset and family of generative models for generating Minecraft worlds at block resolution. Our data comprises billions of high-quality and carefully-balanced cubes from procedurally generated Minecraft terrain and human-authored maps, which we use to study discrete and continuous 3D diffusion models for biome-conditioned chunk generation. When trained with our data, we show that both approaches can generate high-fidelity chunks of the game world, and that the discrete masked diffusion formulation gives us inpainting , outpaining , and user-defined block conditioned generation as a free byproduct of the training objective. This enables players and creators to mold the world around them by generating structures, terrain, and maps that are immediately editable and playable.

Why Minecraft?

Generative AI has made incredible progress in the fields of image, video, and text generation. Despite success in these modalities, the interactive 3D worlds of video games have received much less research attention. We aim to close this gap by releasing a dataset based on one of the most successful and popular video games, Minecraft.

Minecraft is the best-selling video game in history, and a favorite testbed for a variety of AI research. The 3D world of Minecraft is made up of voxels: discrete, compositional building blocks like sand, stone, and water. These semantically meaningful block types provide a perfect abstraction for generative modeling: each piece of the game world is simply a collection of block tokens that come together to create a simulacrum of the natural world. These collections of blocks, or "chunks", capture anything from winding rivers and snow-capped mountains to medieval city centers and futuristic sci-fi worlds. The compact block-based abstraction and nearly infinite expressive range make Minecraft an attractive substrate to study generative systems.

Central to Minecraft is its procedural content generation system, which generates a massive, unique world for each player. These worlds are split into biomes like desert, forest, and ocean, creating a diverse set of environments to explore. Players interact with the world by destroying and placing blocks, creating structures, and terraforming the world as they play. We seek to explore both sides of Minecraft in our work: the natural world generated by algorithmic systems, and human-built structures that capture human creativity in this voxel world.

The Dream-Cubed Dataset

The Dream-Cubed dataset contains both procedurally generated and human-authored Minecraft chunks. We create automated data collection and curation pipelines, collecting 1.8+ million 323 chunks of procedurally generated Minecraft worlds, and 200,00+ chunks from human-authored maps, comprising over 30 billion total blocks.

Minecraft's core procedural content generation provides an endless supply of data. We collect chunks from the surface level of the world and assign a class label based on the biome they are in, creating a dataset for biome-conditional generation. While this data is functionally infinite, biomes are not evenly distributed in generated worlds. Villages, for example, are both the most structurally complex and the least common areas of the game world. To overcome this data balancing problem, we set up targeted collection pipelines to identify and collect specific biomes to ensure adequate representation, which we show can have a very significant impact on downstream models.

Human data, on the other hand, is scarce. There is a rich community centered around the content that players create in their Minecraft worlds. Popular third-party websites like Planet Minecraft allow users to share their creations with others, and some professionals earn income by selling their hand-crafted maps. These maps pose a much deeper challenge for generative modeling, using a larger vocabulary of blocks and containing much more complex structures and environments. We reached out to one such professional map creator, who granted permission to use several of their large and high-quality maps for scraping. We release all our data and models in accordance with the map creators for the larger research and Minecraft community.

Examples from the six human-made maps in the Dream-Cubed dataset. Each map depicts a large-scale city landscape with diverse architectural styles, highlighting the larger block vocabulary and structural complexity found in this data.

Training Models on Blocks

Each Minecraft block is categorical, representing a single block type from a vocabulary of thousands. A natural question is: what kind of generative model is best suited for this data? Continuous diffusion models...