Continuous diffusion language models – Sander Dieleman

Your browser is quite old! Why not upgrade to a different browser to better enjoy this site?

A flurry of recent activity in the space of continuous diffusion models for language , after a few years of relative dormancy, suggests that this approach is making something of a comeback. Fully discrete diffusion methods had largely supplanted earlier attempts to make continuous diffusion work for language, but the tide is starting to turn. In this post, I want to take a closer look at what’s going on, and why it is happening now.

The recent influx of new research in this space inspired me to write up some of my thoughts. I have written about diffusion language models before, so this mainly serves as an update to cover everything that’s happened since then. This will be a fairly subjective account – other perspectives and dissenting opinions are very welcome in the comments and elsewhere! I’ll discuss some technical aspects of continuous diffusion for language later on, but first, some historical context.

Challenging the autoregressive hegemony

Modern language models are, by and large, autoregressive : they generate sequences one token at a time. This is a natural decomposition of a difficult generation task into smaller, easier sequential steps. All steps are instances of the same underlying task (predict a token given preceding tokens), which enables parameter sharing across the sequence dimension. In spite of this inherently sequential generative process, the Transformer architecture1 admits efficient parallel training across all sequence positions using teacher forcing2. This has turned out to be an extremely scalable recipe3, which has brought us large language models (LLMs).

However, autoregression is not the only way to construct an iterative generative process for sequences. Inspired by early successes in the audiovisual domain, researchers sought to apply diffusion to language generation instead. Rather than generating a sequence one element at a time, the generative process of diffusion models is defined by reversing a corruption process, which gradually destroys information. The canonical way to do this is to add Gaussian noise little by little, until it completely overpowers the signal.

2021: early discrete diffusion models

After early successes in image generation in 20194 and 20205 6, the first attempts to apply this idea to language arrived in 2021, and involved replacing a continuous corruption process with a discrete one to enable modelling of categorical data: multinomial diffusion7, D3PM8 and SUNDAE9.

Back then, the dominance of autoregression was not as well-established as it is today: GPT-310 had turned some heads, but the ‘ChatGPT moment’ wouldn’t come until late 2022. At the time, discrete diffusion seemed to address some real theoretical flaws in the autoregressive paradigm, like exposure bias due to teacher forcing and the relative difficulty of applying it to infilling and constrained generation tasks. Note that there had been some exploration of non-autoregressive and any-order autoregressive approaches in the preceding years11 12 (especially for machine translation13 14), but not yet from a diffusion perspective.

2022: continuous diffusion for discrete data

In 2022, several attempts to apply continuous diffusion to language modelling appeared, starting with Diffusion-LM15. This approach addresses the incompatibility between categorical data and corruption with Gaussian noise in a different way: simply represent the discrete categories with continuous embedding vectors, which are perfectly amenable to Gaussian noise corruption. That way, the Gaussian diffusion mechanism, which works so well for images, can be applied without any changes.

Diffusion-LM touted the advantages of this alternative generative paradigm for controllable text generation in particular. In the last few months of 2022, quite a few other papers using variations of this approach were published, including DiffuSeq16, SSD-LM17, Difformer18, SeqDiffuSeq19, GENIE20, LD4LG21 and also two papers that I worked on: self-conditioned embedding diffusion (SED)22 and continuous diffusion for categorical data (CDCD)23.

At the time, the allure of these continuous methods was that they could benefit from all the insights, tools and machinery that were being discovered and developed for continuous diffusion, as it completely took over audiovisual generation. For example, applying some of the sampling and distillation techniques developed for continuous diffusion models to discrete diffusion was often much less straightforward, or even downright impossible.

Late 2023: the continuous extinction

Then, something interesting happened: after 2023, virtually all new research in this space used discrete diffusion , and continuous diffusion for language went extinct. A diagram from a 2025 survey paper24 about diffusion language models clearly shows this:

New survey on...