Introduction

If you are reading this post, chances are the recent hype and excitement around continuous diffusion language models (CDLMs), and diffusion language models in general, have caught your fancy. If that's the case, welcome, welcome! It's always so exciting to see new folks interested in this field.

When I first started discussing CDLMs with fellow researchers, I found it harder than usual to get on the same page. Even seemingly simple claims could be surprisingly confusing. Some of that confusion is understandable in a fast-moving field. Some of it, I suspect, comes from marketing getting a little carried away. Either way, I hope this post saves you from some of the confusion I went through and makes your own journey a little more comfortable.

From autoregression to diffusion

Before we get into diffusion, let's take a step back and look at how most language models generate text. In an industry constantly in flux, GPT-style Transformers are still king. They follow a familiar autoregressive recipe, predicting the next token, appending it to the sequence, and repeating the process. This approach forms the foundation of most leading language models today. But heavy hangs the head that wears the crown. For all its success, this recipe has its limitations, prompting researchers to explore other ways to generate text.

One particularly promising direction is diffusion, which offers another way to organize generation. During training, we corrupt examples of real data and teach a model to reverse that corruption. To generate something new, we start from noise and repeatedly apply the learned model to refine it. Diffusion models have already proven themselves in the vision-domain, and the hope is that this success will transfer.

For language, there are two broad approaches. Discrete diffusion language models (DDLMs) corrupt sequences of categorical tokens, often by replacing tokens with a special mask symbol. Continuous diffusion language models (CDLMs) instead corrupt continuous representations of text, typically by adding Gaussian noise to token embeddings. Generation then refines those noisy representations into something that can be decoded back into tokens.

In this post, we'll focus on continuous diffusion language models. CDLMs are often associated with Transformer architectures, bidirectional attention, parallel generation, and promises of faster inference, but how these ideas fit together isn't always clear. We'll start with a quick diffusion primer, then unpack what defines a CDLM, which features are simply design choices, and how parallel generation looks in practice.

A quick primer on diffusion

Before we get into the weeds, let's briefly look at how diffusion models work. Readers already familiar with the basics should feel free to skip this section.

Images provide a natural starting point. Autoregressive Transformers may dominate language modeling, but diffusion models have been the big dog in image generation.

Imagine taking a photograph and gradually adding random noise. At first, the image remains recognizable. As more noise is introduced, its details become harder to see until little remains but noise. This is the forward process, a predefined procedure for gradually corrupting data. Mathematically, a noisy image can be expressed as

xt=αtx0+σtε,ε∼𝒩(0,I),x_t = \alpha_t x_0 + \sigma_t \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0,I),

where xtx_t is the noisy image at stage tt, x0x_0 is the original image, and ε\varepsilon is sampled Gaussian noise. The coefficients αt\alpha_t and σt\sigma_t control how much of the original image and noise are mixed together.

Four stages show a dog photograph becoming increasingly obscured by Gaussian noise.
An image progressively corrupted by Gaussian noise.

Now, if we can learn to reverse this process, then voilà, we can generate new images from noise.

One common way to learn the reverse process is to train a neural network, called the denoiser, to predict the noise added to an image. We give it the noisy image xtx_t and its noise level tt, and ask it to estimate the noise:

ε^=fθ(xt,t).\hat{\varepsilon} = f_\theta(x_t,t).

Because we added the noise ourselves, we know what the network should try to predict. Training across many images and noise levels teaches it to guide the reverse process.

Once the network is trained, we can start with random noise. We use its prediction to take a denoising step, feed the result back in, and repeat. Little by little, an image takes shape. This is the reverse process, illustrated below. We're generating a new image here, not recovering the original photograph.

A schematic sequence shows random noise becoming a newly generated image.
Illustration of the reverse diffusion process. Starting from random noise, successive denoising steps produce a new image. The panels are schematic rather than outputs from an actual sampling run.

Image diffusion models have traditionally used U-Nets. But diffusion itself is not a neural network architecture. It describes a generative process, while the network used to carry it out is a separate design choice. We'll return to this distinction in Section 4, since it is particularly relevant for language models.

The same principle can be applied to language. But there's a catch. Tokens aren't pixels, so what exactly are we supposed to add noise to?

Where does the diffusion actually happen?

For images, adding Gaussian noise gives us a natural way to gradually destroy information. With language, we first have to decide what to corrupt.

Language models represent text using categorical tokens, whose numerical IDs are essentially arbitrary labels. If a tokenizer assigns two tokens IDs of 100 and 101, that doesn't mean they are linguistically closer than tokens with IDs 100 and 20,000. Simply adding Gaussian noise to these IDs wouldn't give us a meaningful way to gradually corrupt text.

There are two broad approaches to this problem.

Discrete diffusion works directly with categorical tokens. In its most familiar form, masked diffusion, tokens are gradually replaced with a special [MASK] symbol. Generation reverses this process by predicting and revealing the missing tokens. Masking isn't the only option, though. Some models replace tokens with randomly chosen vocabulary entries instead. The key point is that the state being corrupted remains a sequence of discrete tokens.

Continuous diffusion takes a different approach. Rather than applying “noise” directly to discrete tokens, we typically represent each token as a vector of numbers, called an embedding. We can then add Gaussian noise to these vectors much like we did with images:

zt=αtE(x)+σtε,ε∼𝒩(0,I),z_t = \alpha_t E(x) + \sigma_t \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0,I),

where E(x)E(x) represents the embeddings of a token sequence xx, and ztz_t is its noisy representation. During generation, the model gradually denoises these vectors until they can be decoded back into tokens. We'll explore the mathematics in greater detail in a later post.

Here is a simplified comparison of the two corruption processes.

Masked discrete diffusionContinuous diffusion
New YorkToken embeddings
↓\downarrow↓\downarrow
[MASK] YorkPartially noisy embeddings
↓\downarrow↓\downarrow
[MASK] [MASK]Gaussian noise
Two illustrative corruption processes. Generation proceeds in the reverse direction, from masks or noise toward text. Tokenization is simplified for illustration.

Now, here's an important subtlety that's easy to miss. Every conventional Transformer already operates on continuous representations internally. GPT-style Transformers turn their input tokens into continuous embeddings. Discrete diffusion models do much the same. Using continuous vectors alone is NOT what makes a model a CDLM.

The key difference is what gets updated during generation. In discrete diffusion, the network's predictions are used to update a sequence of discrete tokens and masks. In continuous diffusion, the noisy vectors themselves are updated until they can be decoded into text. A CDLM denoises continuous vectors instead of discrete tokens. Of course, this doesn't mean they are completely continuous. Their final outputs are still tokens.

You may have noticed that I refer to “noise” when talking about discrete diffusion. Flipping tokens at random is adding noise, but in the same way that the Dirac delta function is a function. All functions are equal, but some functions are more equal than others.

I personally suspect that evolving through a continuous space gives CDLMs a real advantage in expressivity, particularly when generating multiple dependent tokens together. We'll return to this when we look at how tokens are generated together.

Under the hood, isn't it still a Transformer?

By now, you may have noticed the deep, dark secret. Under the hood, many diffusion language models are still Transformers! It's surprisingly common to see diffusion and Transformers discussed as though they were competing architectures, but they aren't describing the same thing. Recall the denoising network fθf_\theta from our image diffusion example. In image diffusion, this network has traditionally been a U-Net, while in language diffusion, it's often a Transformer. Diffusion describes the generative process, while the Transformer is the network used to guide it.

Part of the confusion comes from how closely we've come to associate the Transformer with the GPT recipe. GPT combines the Transformer architecture with causal attention and training to predict the next token. It generates text one token at a time. Diffusion language models can keep much of the same architecture while giving it a different job. A masked DDLM might use the network to predict missing tokens, while a CDLM might use it to denoise continuous representations. Instead of immediately choosing the next token, a diffusion model typically uses repeated predictions to refine its tokens or noisy vectors.

Bidirectional attention and block-autoregressive generation

This brings us to another characteristic commonly associated with diffusion language models, bidirectional attention. When denoising several positions together, why not let them exchange information? Unlike the causal attention used in GPT-style Transformers, bidirectional attention lets each position attend to positions on both its left and right. Of course, this doesn't mean the model can peek at the actual future text. It only has the current tokens, masks, or noisy vectors to work with.

The idea itself is nothing new. BERT was already using bidirectional attention to predict masked tokens long before diffusion language models became popular. And while bidirectional attention is a natural fit for denoising several positions together, it isn't a requirement for CDLMs. In principle, we could even use causal attention and denoise one token at a time before moving to the next. To me, though, that looks more like an ablation experiment than something you'd actually deploy.

We also don't have to denoise an entire sequence at once. A common approach is block-autoregressive generation, where we divide the output into smaller blocks and generate them one after another. Within each block, attention can remain bidirectional, allowing the tokens to be refined together. Each new block uses the completed blocks as context, so generation across blocks is autoregressive.

This makes longer outputs more practical. The model can often save and reuse computation from completed blocks as it generates the next ones.

Generation patterns

Autoregressive

One token at a time
x1x2x3x4x5x6
generate left to right

Whole-sequence
diffusion

All positions, several steps
x1x2x3x4x5x6
refine all positions

Block diffusion

One block at a time
x1x2
refine within block
x3x4
refine within block
x5x6
refine within block
Schematic · steps are illustrative, not a speed comparison.
Three approaches to text generation: token by token, whole-sequence diffusion, and block-autoregressive diffusion. Refinement within a sequence or block may involve multiple denoising steps.

Both DDLMs and CDLMs can use these generation strategies. They describe how generation is organized, while the discrete–continuous distinction concerns what is being denoised. That brings us to another common misunderstanding, and perhaps the hardest one to wrap our heads around. Denoising several tokens together doesn't mean generating them all in a single step.

Predicting many tokens is not the same as generating many tokens

Now this is where we separate marketing from reality. Diffusion language models are often described as being able to generate hundreds of tokens “at once.” That sounds impressive, but what exactly does “at once” mean?

The key distinction is between predicting, updating, and finalizing tokens. By finalizing, I mean committing to the tokens that will actually appear in the output. A model can predict many positions in a single forward pass without finalizing any of them. A masked DDLM, for example, might predict all 100 masked positions but reveal only ten before making another pass. A CDLM might instead update 100 noisy token representations together, refining them over ten passes before converting them into discrete tokens.

So yes, diffusion models can work on many positions in parallel and produce an entire block of tokens together. But that doesn't mean the whole block was generated in a single forward pass. In our CDLM example, each pass updates all 100 token representations at once, but we still need multiple passes to finish generating the block. Parallel token updates and one-step generation are not the same thing.

And even if we could generate a hundred tokens in a single pass, there's another question. How do we make sure those tokens make sense together?

The interesting part isn't just parallelism. It's dependence.

Now we get to what I find particularly interesting about CDLMs. Generating many tokens at once sounds great, but how do we make sure those tokens actually make sense together? Continuous diffusion offers a particularly elegant way of addressing this.

Let's say we want a model to complete the sentence:

I live in ___ ___.

Imagine that the two equally likely completions are New York and Hong Kong. We want the model to choose one of these cities, not something like New Kong or Hong York.

An autoregressive model handles this quite naturally. It chooses the first token, then conditions the second on that choice. If it chooses New, it knows to follow with York. But what if we want to generate both tokens simultaneously? If we sample them independently, the first position has a 50% chance of being New or Hong, while the second has a 50% chance of being York or Kong. Half of our combinations mix up the two cities, even though the probabilities at each individual position are correct.

The problem is that getting the probabilities right for each token doesn't mean getting their combinations right. Mathematically, sampling two tokens independently given the current context or denoising state cc gives us

q(x1,x2|c)=q(x1|c)q(x2|c).q(x_1,x_2\mid c) = q(x_1\mid c)\,q(x_2\mid c).

This is sometimes called the factorization problem. Bidirectional attention alone doesn't solve it. Both positions can use information from the whole sequence, but if we then draw each token independently, we can still end up with mismatched choices.

In many masked DDLMs, tokens revealed in the same step are sampled independently given the current sequence. Later steps can build on those choices. Once New has been chosen, for example, the next step can use that information to predict York. This allows them to capture dependencies over time. The challenge comes when we want to generate many dependent tokens in very few steps.

CDLMs offer another way to build these dependencies. A CDLM starts with noisy continuous vectors and gradually refines them together. The denoising network can use information from all positions, allowing their representations to influence one another before any final token decisions are made.

What matters here is what the model carries from one step to the next. In a standard masked DDLM, each position is either masked or represented by a discrete token. A CDLM instead carries a continuous vector from one denoising step to the next. That vector can change gradually and influence the other positions without first becoming a token. In our example, the two representations can work toward a consistent city name while both final token choices remain open.

Even if the initial noise at each position was sampled independently, the final tokens don't need to be independent. The model transforms those noisy vectors together, so the same initial randomness can influence several final token choices. Independent starting noise can become a coordinated output.

Discrete models can coordinate token choices too. Some use samplers that choose related tokens together, instead of drawing each one independently. The appeal of continuous diffusion is that it can build this coordination through gradual changes to the vectors themselves. How many steps it needs to do that well is a separate question.

For me, the real promise of CDLMs isn't simply generating more tokens in parallel. It's generating dependent tokens together, without having to make every decision one by one.

So what actually makes a model a CDLM?

With all this out of the way, the core definition is really simple:

A continuous diffusion language model generates text by denoising continuous vectors. These vectors are then decoded into tokens.

That's really the defining characteristic. Everything else we've discussed is either a common design choice or a potential advantage, rather than something inherent to CDLMs.

Here's a quick checklist:

CharacteristicWhat to remember
Continuous generative stateEssential. This defines a CDLM.
Transformer architectureCommon, not required.
Bidirectional attentionCommon, not required.
Parallel multi-token updatesCommon, not required.
Whole-sequence generationOptional. Block-autoregressive generation is also common.
Joint generation of dependent tokensA key attraction, but not guaranteed.
Faster than autoregressive modelsNot necessarily. Denoising has a cost.

I hope this has made CDLMs a little less mysterious and a little less intimidating. These were all things I found confusing myself when I first started studying them. For those interested in going deeper, we'll explore the technical details, history, and design choices in future posts.

XOXO, K


Further reading