Diffusion, for Images and for Text: Learn to Undo a Little Noise

VAEs and GANs ended with two ways to turn noise into data in a single jump: a VAE’s decoder, which put half its samples where there was no data, and a GAN’s generator, which was sharp but dropped modes and sometimes didn’t train at all. Diffusion models take many small jumps instead. Corrupt the data a little at a time until nothing is left but noise; train one network to undo a single step; then start from noise and apply it again and again. Each step is a plain regression problem, with no adversary and no bound to keep tight, and that stability is why diffusion replaced GANs for images.

The same idea works for text, with masking in place of Gaussian noise, and it changes something autoregression (What “Generative” Means) takes for granted: a diffusion language model can write many tokens at once, in any order. This post covers both, and measures what each step count, guidance setting and degree of parallelism costs.

Table of contents

The forward process

Take a data point $x_0$ and add a little Gaussian noise, $T$ times:

where the noise schedule $\beta_1, \dots, \beta_T$ says how much noise each step adds. Scaling by $\sqrt{1 - \beta_t}$ keeps the total variance from growing (exactly, if the data has unit variance; data is usually normalized to roughly that): at the end, $x_T$ is very nearly a standard Gaussian, whatever $x_0$ was.

Nothing here is learned, and it has a closed form. A sum of independent Gaussians is Gaussian, so $t$ steps compose into one. Write $\alpha_t = 1 - \beta_t$ and $\bar\alpha_t = \alpha_1 \alpha_2 \cdots \alpha_t$:

Training never has to simulate the chain: it can jump to any step $t$ in one line. $\bar\alpha_t$ is the share of the original signal’s variance left at step $t$.

The schedule, worked

Ho et al. (2020) used $T = 1000$ steps with $\beta_t$ rising linearly from $10^{-4}$ to $0.02$. The first lines of the experiment script compute what that does to the signal:

linear schedule, beta from 1e-4 to 0.02 over T = 1000 steps
    t    abar_t  signal sqrt(abar)  noise sqrt(1-abar)       SNR
    1   0.99990             0.9999              0.0100      9999
   10   0.99811             0.9991              0.0435     526.8
  100   0.89702             0.9471              0.3209      8.71
  250   0.52409             0.7239              0.6899     1.101
  500   0.07859             0.2803              0.9599   0.08529
  750   0.00335             0.0579              0.9983  0.003362
 1000   0.00004             0.0064              1.0000 4.036e-05

The signal-to-noise ratio, $\bar\alpha_t / (1 - \bar\alpha_t)$, falls from about 10,000 after the first step to 1 at step 250 and $4 \times 10^{-5}$ at the end. Two things follow. The last few hundred steps are nearly indistinguishable, pure noise, and the schedule spends many of its steps there; the cosine schedule (Nichol and Dhariwal, 2021) was designed to destroy information more evenly. And $\bar\alpha_T$ is small but not zero: the final step still holds 0.6% of the signal’s scale, so sampling from a pure Gaussian starts the reverse process slightly off.

Drag the slider to see it on the eight-mode data of VAEs and GANs. The left panel is this formula applied to the data; the right is the trained model from the experiment below, run backwards from pure noise:

The loss, from the ELBO

The model is the reverse chain: $p_\theta(x_{t-1} \mid x_t)$, a Gaussian whose mean a network computes, started from $p(x_T) = \mathcal N(0, I)$. This is a latent-variable model like the VAE, with $x_1, \dots, x_T$ as the latents, and the same Jensen argument as VAEs and GANs gives an ELBO. What makes it tractable is that the forward process is fixed, so the “encoder” has no parameters, and its posterior given the data has a closed form:

where $\varepsilon$ is the noise that produced $x_t$ from $x_0$, and $\tilde\beta_t = \beta_t (1 - \bar\alpha_{t-1}) / (1 - \bar\alpha_t)$. The ELBO splits into one KL term per step, each between this Gaussian and the model’s. The model’s variance is fixed too (to $\beta_t$ or $\tilde\beta_t$), and between two Gaussians with fixed variances the KL is a squared distance between their means, plus a constant. So if the model’s mean takes the same form, with a network $\varepsilon_\theta(x_t, t)$ in place of the true $\varepsilon$, each term is a weighted squared error between $\varepsilon_\theta$ and $\varepsilon$. (Two end terms remain: one for decoding $x_1$ into data, and one comparing $x_T$ with the prior, which has no parameters.) Ho et al. found that dropping the weights gave better samples, at the cost of a slightly worse likelihood bound, which gives the $\varepsilon$-prediction loss:

Pick a data point, a random step and random noise; noise the data in one line; ask the network which noise was added. That’s the whole training loop. It is a mean-squared-error regression, which is why diffusion trains as reliably as any supervised model. (Predicting $\varepsilon$ carries the same information as predicting $x_0$, since either determines the other given $x_t$, or as predicting the gradient of the log-density of the noised data, its score, which is $-\varepsilon / \sqrt{1 - \bar\alpha_t}$ in expectation: that’s how Song and Ermon’s score-based models arrived at the same method from another direction. As losses they differ: squared error on $x_0$ is squared error on $\varepsilon$ divided by the signal-to-noise ratio, and the choice of weighting matters in practice.)

Sampling: DDPM and DDIM

DDPM sampling runs the learned chain: start from $x_T \sim \mathcal N(0, I)$, and for $t = T, \dots, 1$ compute the mean above with $\varepsilon_\theta$ and add fresh noise of variance $\beta_t$. A thousand network evaluations per sample.

DDIM (Song, Meng and Ermon, 2021) observed that the training loss only involves $q(x_t \mid x_0)$, the one-jump marginals, not the step-by-step chain. Many reverse processes share those marginals, including a deterministic one: estimate the clean point, $\hat x_0 = (x_t - \sqrt{1 - \bar\alpha_t}\, \varepsilon_\theta) / \sqrt{\bar\alpha_t}$, then move straight to an earlier level $s$, reusing the same noise estimate:

Because $s$ can be any earlier step, DDIM can skip: sample in 50 steps with a network trained for 1,000, without retraining. In the limit of many small steps, DDIM is following an ordinary differential equation that carries noise to data (the probability-flow ODE), and each DDIM step is one step of the simplest numerical method for it, Euler’s. That view is where faster samplers come from: better ODE solvers take fewer, larger steps for the same accuracy.

One model, many step counts

The experiment: VAEs and GANs’s eight modes, scored the same way, with an $\varepsilon$-network about the size of VAEs and GANs’s (three hidden layers of 256), trained for 20,000 steps on three seeds. It’s trained with the mode’s label as an extra input that is dropped 10% of the time, for the guidance experiment below; dropping the label gives unconditional samples. The script is diff2d.py. It saves the trained networks to nets.pt, and the later scripts load both by name. It runs on the CPU; on an Apple M4 laptop this post’s code takes about 5 minutes in all:

def train(seed, steps=20000):
    g = torch.Generator().manual_seed(seed); torch.manual_seed(seed)
    net = Eps(); opt = torch.optim.Adam(net.parameters(), 1e-3)
    sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, steps)
    for _ in range(steps):
        x0, c = data(512, g)
        c = torch.where(torch.rand(512, generator=g) < 0.1, 8, c)             # drop the label 10% of the time
        t = torch.randint(0, T, (512,), generator=g)
        eps = torch.randn(512, 2, generator=g)
        a = abar[t][:, None]
        loss = ((net(a.sqrt() * x0 + (1 - a).sqrt() * eps, t, c) - eps) ** 2).mean()   # the DDPM loss
        opt.zero_grad(); loss.backward(); opt.step(); sched.step()
    return net

@torch.no_grad()
def guided(net, x, t, c, w):
    """Classifier-free guidance: eps_uncond + w (eps_cond - eps_uncond).
    w = 1 is the plain conditional model; c = 8 means unconditional."""
    tt = torch.full((len(x),), t)
    if w == 1 or (c == 8).all():
        return net(x, tt, c)
    e_u = net(x, tt, torch.full_like(c, 8))
    return e_u + w * (net(x, tt, c) - e_u)

@torch.no_grad()
def ddpm(net, n, c=8, w=1.0):
    """Ancestral sampling, all T steps: x_{t-1} = mean + sqrt(beta_t) z."""
    x, c = torch.randn(n, 2), torch.full((n,), c) if isinstance(c, int) else c
    for t in reversed(range(T)):
        e = guided(net, x, t, c, w)
        x = (x - beta[t].float() / (1 - abar[t]).sqrt() * e) / (1 - beta[t].float()).sqrt()
        if t > 0:
            x = x + beta[t].float().sqrt() * torch.randn_like(x)
    return x

@torch.no_grad()
def ddim(net, n, steps, c=8, w=1.0):
    """DDIM with eta = 0 (Song et al., 2021): deterministic, and free to skip
    steps. Predict x_0 from the noise estimate, then jump to the next t."""
    x, c = torch.randn(n, 2), torch.full((n,), c) if isinstance(c, int) else c
    ts = torch.linspace(T - 1, 0, steps).round().long().tolist()
    for i, t in enumerate(ts):
        e = guided(net, x, t, c, w)
        x0 = (x - (1 - abar[t]).sqrt() * e) / abar[t].sqrt()
        a_next = abar[ts[i + 1]] if i + 1 < len(ts) else torch.tensor(1.0)
        x = a_next.sqrt() * x0 + (1 - a_next).sqrt() * e
    return x

Full script: diff2d.py

train is the DDPM loss: noise a clean point to a random step $t$ and regress the noise. ddpm and ddim are the two samplers, and guided mixes the conditional and unconditional predictions. The rest of diff2d.py defines the eight-mode data and its score, the schedule, and the network Eps, an MLP on the point, a sinusoidal embedding of $t$ and a learned embedding of the label, then trains three seeds, prints the tables and saves nets.pt.

unconditional samples (2,000 per run); modes covered, then share near a mode, per seed
DDPM, 1000 steps     8  8  8     84%   85%   85%
DDIM, 100 steps      8  8  8     83%   82%   81%
DDIM, 50 steps       8  8  8     81%   80%   80%
DDIM, 20 steps       8  8  8     76%   73%   73%
DDIM, 10 steps       8  8  8     58%   55%   51%
DDIM, 5 steps        0  0  1      3%    3%    3%
DDIM, 2 steps        0  1  0      2%    2%    2%

With all 1,000 steps, every seed covers all eight modes with 84–85% of samples near one: the coverage of the VAE in VAEs and GANs with the sharpness of its GAN, from a loss with neither one’s failure modes. (For reference, a perfect sampler of this data would put 98.9% within 3 standard deviations, so there’s room left.) DDIM holds up to about 20 steps (73–76%), degrades at 10 (51–58%, still every mode), and at 5 steps produces almost nothing near a mode.

That last row turned out to measure where the steps fell, as much as how many there were. Evenly spaced from 999 down to 0, five steps visit $t = 999, 749, 500, 250, 0$: the last one, at $t = 0$, has almost no noise left to remove, so in effect one jump goes from $t = 250$ straight to clean data, landing on the average of the nearby modes. The same models, sampled with other spacings:

@torch.no_grad()
def ddim_at(net, n, ts):
    """diff2d.py's ddim(), visiting the given timesteps (descending)."""
    x, c = torch.randn(n, 2), torch.full((n,), 8)
    for i, t in enumerate(ts):
        e = ns["guided"](net, x, t, c, 1.0)
        x0 = (x - (1 - abar[t]).sqrt() * e) / abar[t].sqrt()
        a_next = abar[ts[i + 1]] if i + 1 < len(ts) else torch.tensor(1.0)
        x = a_next.sqrt() * x0 + (1 - a_next).sqrt() * e
    return x

spacings = {
    "even, ending at t = 0 (diff2d.py)": lambda k: torch.linspace(999, 0, k).round().long().tolist(),
    "even, ending at t = 999/k": lambda k: [round(999 - i * 1000 / k) for i in range(k)],
    "quadratic (DDIM paper)": lambda k: sorted({round(999 * (1 - i / k) ** 2) for i in range(k)}, reverse=True),
}

Full script: ddim_spacing.py

ddim_spacing.py loads the definitions from diff2d.py and the trained networks from nets.pt, then samples with each spacing at 5, 10 and 20 steps.

unconditional, 2,000 samples per seed: share near a mode (modes covered)
  5 steps, even, ending at t = 0 (diff2d.py)    3% (0)    3% (0)    3% (1)   t = [999, 749, 500, 250, 0]
  5 steps, even, ending at t = 999/k           17% (8)   13% (7)   12% (8)   t = [999, 799, 599, 399, 199]
  5 steps, quadratic (DDIM paper)              50% (8)   42% (8)   39% (8)   t = [999, 639, 360, 160, 40]
 10 steps, even, ending at t = 0 (diff2d.py)   58% (8)   55% (8)   51% (8)
 10 steps, even, ending at t = 999/k           61% (8)   58% (8)   55% (8)
 10 steps, quadratic (DDIM paper)              67% (8)   65% (8)   63% (8)
 20 steps, even, ending at t = 0 (diff2d.py)   76% (8)   73% (8)   73% (8)
 20 steps, even, ending at t = 999/k           76% (8)   74% (8)   75% (8)
 20 steps, quadratic (DDIM paper)              77% (8)   74% (8)   74% (8)

Quadratic spacing, which the DDIM paper itself used, puts more steps where the noise is low and the structure forms, and at 5 steps gives 39–50% near a mode, every mode covered, against 3%. By 20 steps the choice hardly matters. The schedule of sampling steps is as much a design choice as the number. Much of the work since DDIM has been about pushing the usable step count lower: better spacings and noise schedules, higher-order ODE solvers, and distillation into one to four steps (VAEs and GANs’s adversarial distillation is one; consistency models, which learn to jump from any noise level straight to the clean point, are another).

Classifier-free guidance

To generate a particular class or match a prompt, give the network the condition $c$ as an input: $\varepsilon_\theta(x_t, t, c)$. In practice the conditional model alone follows its condition too loosely, so diffusion models use classifier-free guidance (Ho and Salimans, 2021): train one network both with the condition and without it (by dropping $c$ some of the time), then sample with

At $w = 1$ this is the conditional model; at $w = 0$, the unconditional one; $w > 1$ pushes past the conditional model, in the direction that makes the sample more like $c$ and less like data in general. Text-to-image systems typically use $w$ well above 1, trading diversity for prompt adherence. (This is the convention of most libraries. Ho and Salimans’ paper writes the same thing as $(1 + w)\,\varepsilon_\theta(x_t, t, c) - w\,\varepsilon_\theta(x_t, t, \varnothing)$, so their $w = 0$ is the conditional model; check which one a library means.) The cost is two network evaluations per step, which guidance distillation removes by training a student to produce the guided prediction in one evaluation.

On the eight modes, asking for each mode 250 times:

classifier-free guidance, DDIM 50 steps, 250 samples asked for each mode (2,000 per run)
   w   landed on the mode asked for   share near any mode
   0                10%   10%   11%       83%   81%   81%
   1               100%  100%  100%      100%  100%  100%
   3                89%   93%   89%       89%   93%   89%
   7                 4%    0%   12%        4%    0%   12%

Unguided ($w = 0$), about 10% of samples land on the mode asked for: the 1/8 chance times the 81–83% that land on any mode. The plain conditional model ($w = 1$) lands on the requested mode every time. More guidance then makes things worse: 89–93% at $w = 3$, and at $w = 7$ almost nothing lands on any mode. Where do the guided samples go?

import math
import torch
# Where guided samples go: for each sample, its distance from the center
# of the ring (the true modes sit at radius 2) and its spread around the
# mode it was asked for (the data's own spread is 0.05 per coordinate).
src = open("diff2d.py").read().split("nets = [train(s)")[0]
ns = {}; exec(src.replace("print(", "(lambda *a, **k: None)("), ns)
nets = []
for sd in torch.load("nets.pt"):
    m = ns["Eps"](); m.load_state_dict(sd); nets.append(m)
asked = torch.arange(8).repeat_interleave(250)
print("DDIM 50 steps, 250 samples per mode, all three seeds pooled")
print(f"{'w':>3s} {'median radius':>14s} {'spread around the mode asked for':>33s}")
for w in (1.0, 3.0, 7.0):
    torch.manual_seed(200)
    xs = torch.cat([ns["ddim"](m, 2000, 50, c=asked, w=w) for m in nets])
    target = ns["centers"][asked.repeat(3)]
    print(f"{w:3.0f} {xs.norm(dim=1).median():14.2f} {(xs - target).std(0).mean():33.3f}")

Full script: guidance_radius.py

DDIM 50 steps, 250 samples per mode, all three seeds pooled
  w  median radius  spread around the mode asked for
  1           2.00                             0.032
  3           2.06                             0.081
  7           3.98                             4.737

Outward. At $w = 3$ the median sample sits slightly beyond the ring, and at $w = 7$ at nearly twice its radius: still in the right direction, but overshooting. Guidance extrapolates from the unconditional prediction through the conditional one, and the result is no longer the noise estimate of any actual noised distribution, so nothing keeps the trajectory on the data. Image samplers counter the same effect by clipping or rescaling the predicted clean image at each step, and it’s why high guidance gives oversaturated, distorted pictures. Note also the $w = 1$ row: its samples spread 0.032 around their mode, tighter than the data’s 0.05. Landing on the right mode every time isn’t the same as matching the distribution. On real images, where a prompt leaves much undetermined, moderate guidance helps prompt adherence; it is a knob to tune per model, not a free improvement.

Latent diffusion

Running a thousand steps on 512×512×3 pixels is expensive, and most of those pixels’ detail is imperceptible. Latent diffusion (Rombach et al., 2022) runs the diffusion on a compressed representation instead: a VAE, trained first with a perceptual loss (distances between a pretrained network’s features rather than between pixels) and an adversarial one, encodes each image to a 64×64×4 latent grid, 48 times fewer numbers. The diffusion model learns to generate latents, with the text prompt fed in through cross-attention: attention whose queries come from the image latents and whose keys and values come from the text encoder’s outputs. The VAE’s decoder turns the final latent into pixels. This is VAEs and GANs’s VAE in its modern role: not a sampler, but the space a better sampler works in.

Stable Diffusion’s first versions are built this way, with a U-Net predicting the noise: a convolutional network that downsamples the image in stages and upsamples it back, with skip connections between matching resolutions. Newer systems replace it with a Transformer over patches of the latent (DiT). Their training objective has moved on too. Many predict a combination of noise and signal ($v$-prediction) instead of $\varepsilon$, and the newest image models (Stable Diffusion 3, FLUX) train with flow matching or rectified flow: learn the velocity of a straight-line path from noise to data, which makes the sampling ODE nearly straight and easier to solve in few steps. The loss is still a regression; the path from noise to data is what changed.

Diffusion for text: masking instead of noise

Text is discrete, so there is no small Gaussian step to take. The discrete analogue (Austin et al., 2021, D3PM) replaces adding noise with corrupting tokens, and the variant that works best is the simplest: masking. The forward process replaces each token with [MASK] independently; at time $t \in [0, 1]$ each token is masked with probability $t$, so $t = 1$ is a fully masked sequence. The reverse model is a bidirectional Transformer, like BERT, that predicts the original token at every masked position.

The ELBO works out to something simple (Sahoo et al., 2024, MDLM; Shi et al., 2024): cross-entropy on the masked tokens, weighted by $1/t$, averaged over $t$:

It’s BERT’s masked-language-model loss, with the masking rate drawn at random for each sequence (anywhere up to 100%) instead of fixed at 15%, and that weighting makes it a bound on the log-likelihood. (The toy below keeps $t \ge 1/8$, so at least one of its eight tokens is masked on average, and divides by the length; neither changes what it learns.) LLaDA (Nie et al., 2025) trained an 8B model this way and reported results comparable to autoregressive models of similar size on several benchmarks.

Generation starts from all [MASK]s. Each step, the model predicts every masked position, and the sampler fills in some of them, by a fixed schedule or the ones the model is most confident about, leaving the rest masked for later. Positions filled in the same step are sampled independently, each from its own predicted distribution. That’s the source of the speed, and of the problem.

What parallel decoding gets wrong

A toy language makes the problem exact. Its sentences are eight digits in arithmetic progression mod 10, like 3 6 9 2 5 8 1 4: 90 valid sequences, in which two adjacent digits pin down the rest. Train a small masked diffusion model on them, then generate with 1, 2, 4 or all 8 tokens unmasked per step:

def train(seed, steps=10000):
    """The masked-diffusion loss (MDLM, Sahoo et al. 2024): draw a masking
    rate t, mask each token with probability t, and score the model's
    cross-entropy on the masked tokens, weighted by 1/t."""
    torch.manual_seed(seed)
    net = Denoiser(); opt = torch.optim.Adam(net.parameters(), 1e-3)
    for _ in range(steps):
        x0 = lang[torch.randint(0, len(lang), (256,))]
        t = torch.rand(256, 1).clamp(min=1 / L)
        m = torch.rand(256, L) < t
        logits = net(torch.where(m, MASK, x0))
        ce = nn.functional.cross_entropy(logits.transpose(1, 2), x0, reduction="none")
        loss = ((ce * m) / t).sum(1).mean() / L
        opt.zero_grad(); loss.backward(); opt.step()
    return net

@torch.no_grad()
def sample(net, n, k, order):
    """Start fully masked; each step unmasks k positions (k can be a list,
    one entry per step), sampling each from the model's distribution for
    that position, independently of the others unmasked in the same step."""
    x = torch.full((n, L), MASK)
    for k in (k if isinstance(k, list) else [k] * (L // k)):
        probs = net(x).softmax(-1)
        draw = torch.distributions.Categorical(probs).sample()
        free = x == MASK
        if order == "left to right":
            key = -torch.arange(L).float().expand(n, L)
        elif order == "random":
            key = torch.rand(n, L)
        else:                                               # "most confident": highest top probability first
            key = probs.max(-1).values
        pick = torch.where(free, key, -float("inf")).topk(k, -1).indices
        x.scatter_(1, pick, draw.gather(1, pick))
    return x

Full script: masked_diffusion.py

train is the masked-diffusion loss and sample the parallel decoder. The rest of masked_diffusion.py builds the 90 sequences and the Denoiser, a small bidirectional Transformer, computes the exact first-step chance, and prints the table, the two-then-six schedule and the count behind random order’s failure.

90 valid sequences of 8 digits; 3 models (seeds), 2,000 samples each

tokens per step  steps           order   valid samples, per seed     distinct valid (of 90)   first step, exact
              1      8   left to right    99.1%   98.5%   98.7%       90   90   90            100.00%
              1      8          random    99.1%   98.9%   98.3%       90   90   90            100.00%
              1      8  most confident    99.9%   99.8%   99.9%       90   90   90                   
              2      4   left to right    90.1%   89.5%   89.2%       90   90   90             90.00%
              2      4          random    58.1%   58.8%   58.2%       90   90   90             65.36%
              2      4  most confident    30.4%   88.4%   89.2%       88   90   90                   
              4      2   left to right     1.1%    0.9%    1.0%       22   19   18              0.90%
              4      2          random     1.1%    0.9%    0.5%       16   18    9              0.89%
              4      2  most confident     1.1%    1.0%    1.0%       19   16   19                   
              8      1     all at once     0.0%    0.0%    0.0%        0    0    0              0.00%
   1, 1, then 6      3   left to right    98.8%   98.0%   98.1%       90   90   90
   1, 1, then 6      3          random    55.1%   56.5%   54.8%       90   90   90
digits 1 apart: 9 of the 9 possible differences fix the step
digits 2 apart: 1 of the 5 possible differences fix the step
digits 3 apart: 9 of the 9 possible differences fix the step
digits 4 apart: 1 of the 5 possible differences fix the step
digits 5 apart: 0 of the 2 possible differences fix the step
digits 6 apart: 1 of the 5 possible differences fix the step
digits 7 apart: 9 of the 9 possible differences fix the step

One token per step, in any order, the model writes a valid sequence 98–100% of the time and produces all 90 (96–100% on a rerun on an NVIDIA RTX 3090, its saved outputs). That is ordinary sampling, one conditional at a time; the small shortfall is the model’s own error. Two tokens per step, left to right, gives 89–90% valid, and the last column says why exactly. The first step fills two positions from a fully masked sequence, where the right distribution for each position on its own is uniform over the ten digits. Sampled independently, the two digits are valid together unless they’re equal (a step of 0 isn’t allowed): 90 of the 100 pairs. Every later step is conditioned on digits already fixed and almost never fails. The measured 89–90% is the exact 90%, less the model’s error.

Random positions do worse, for two reasons. The first step: two digits an even distance apart must differ by an even amount (a gap of $g$ adds $g d$ mod 10), so at gaps of 2, 4 or 6 half the pairs of digits can’t both be right, and at a gap of 5 only two differences can; averaged over random positions, 65% of first steps are consistent. Then later steps: two consistent digits pin the sequence down only if their gap shares no factor with 10 (1, 3 or 7 positions). At other gaps several sequences remain possible, and the next parallel step can mix them. That’s the further drop to the measured 58%, and the “1, 1, then 6” row in random order shows the second effect on its own: two digits placed one at a time are always consistent, yet sampling the other six together still fails 45% of the time, when the two happened to land at a gap that leaves the step ambiguous.

Four tokens per step falls to about 1%, again matching the exact 0.9% of valid four-digit patterns. (The few valid outputs, about 20 of each 2,000, can’t cover more than a handful of the 90 sequences.) All eight at once: none. Choosing the “most confident” positions first, a common heuristic, is decided per model here, not per sample (30%, 88% and 89% valid for the three seeds on the M4; 30%, 5% and 89% on the rerun), and the code says why: at the first step every sample is the same fully masked sequence, so the model’s confidences, and the positions chosen, are identical for all 2,000 samples of a model. Which pair that is depends on tiny differences between equally uncertain positions, and the pair was a bad gap for one seed on the M4 and for two on the rerun.

The independence isn’t a flaw in this model; it’s the definition of the parallel step. The model predicts each position’s distribution correctly, and the error comes from sampling them as if they didn’t constrain each other. The cure is in the schedule. Two adjacent digits first, one at a time, then the other six in a single step: 96–99% valid in three steps, about as good as eight steps one at a time. Parallel decoding is safe exactly when what’s already written determines what’s left, which is why practical samplers unmask few tokens early and more later, and why tightly constrained text (code, numbers, a proof) is the hard case for diffusion language models.

The arithmetic: FLOPs and latency

How does a diffusion language model’s cost compare with autoregressive decoding? Take an 8B model generating 512 tokens on one H100 at batch 1:

# Generating L = 512 tokens with an 8B-parameter model on one H100 (989
# TFLOPS dense BF16, 3.35 TB/s), batch 1, ignoring attention's own cost.
# A forward pass costs about 2N FLOPs per position it processes, and reads
# all 2N bytes of BF16 weights once whatever the number of positions.
#   Autoregressive: L passes of 1 position each (the KV cache keeps earlier
#   positions). Each pass is memory-bound.
#   Masked diffusion: S passes over all L positions, because every position
#   can change at every step and a bidirectional model can't cache them.
N, L = 8e9, 512
flops_rate, bw = 989e12, 3.35e12
def step_time(positions):          # the slower of reading the weights and doing the arithmetic
    return max(2 * N / bw, 2 * N * positions / flops_rate)
ar_flops, ar_time = 2 * N * L, L * step_time(1)
print(f"autoregressive: {L} steps, {ar_flops / 1e12:6.1f} TFLOP, at least {ar_time:5.2f} s")
print(f"{'diffusion steps S':>18s} {'tokens per step':>16s} {'TFLOP':>8s} {'x AR FLOPs':>11s} {'time at least':>14s}")
for S in (512, 256, 128, 64, 32, 16):
    f = 2 * N * L * S
    print(f"{S:18d} {L / S:16.0f} {f / 1e12:8.0f} {f / ar_flops:11.0f} {S * step_time(L):12.2f} s")

Full script: flops.py

autoregressive: 512 steps,    8.2 TFLOP, at least  2.45 s
 diffusion steps S  tokens per step    TFLOP  x AR FLOPs  time at least
               512                1     4194         512         4.24 s
               256                2     2097         256         2.12 s
               128                4     1049         128         1.06 s
                64                8      524          64         0.53 s
                32               16      262          32         0.27 s
                16               32      131          16         0.13 s

Diffusion does far more arithmetic: $S$ passes over all $L$ positions against autoregression’s one pass per position, because a bidirectional model can’t cache earlier positions when any of them may still change. At 64 steps that’s 64 times the FLOPs. But autoregression at batch 1 is memory-bound (Small and Specialized Models): 512 steps each waiting 4.8 ms for the weights, with the arithmetic units nearly idle. Diffusion fills them. As long as the number of steps is well below the number of tokens, it can finish sooner while doing more work: 0.53 seconds at 64 steps against 2.45. These are ideal bounds, and real kernels reach only part of peak arithmetic, but at 64 steps the bound is more than four times below autoregression’s. At one token per step, it’s slower.

The advantage is specific to small batches. A server running many sequences at once already keeps the arithmetic busy with autoregression, and then diffusion’s extra FLOPs are simply extra cost. This is the same trade as speculative decoding (Speculative Decoding, Derived), which also spends spare arithmetic to cut the number of sequential steps; the difference is that speculation’s output is exactly the autoregressive model’s, while parallel diffusion decoding changes the distribution, as the toy showed. The FLOPs gap is also narrowing. LLaDA itself samples semi-autoregressively, a block of tokens at a time from left to right, and block diffusion (Arriola et al., 2025) builds that into training: autoregressive across blocks, diffusion within each, so finished blocks can be cached like any prefix. Caching schemes for full-sequence diffusion LMs (Fast-dLLM, for one, with a “block-wise approximate KV Cache”) reuse most keys and values between nearby steps.

Commercial diffusion LMs exist and sell on exactly this. Inception Labs’ Mercury report (2025) says its Mercury Coder models “outperform speed-optimized frontier models by up to 10x on average while maintaining comparable quality”, and Google has shown an experimental Gemini Diffusion. Whether they keep up in quality on long, tightly-constrained outputs (code that has to compile, a proof) is the open question the toy above is a miniature of.

One more connection. Small and Specialized Models’s decision models answer a fixed set of typed questions about a text all at once, which is the one-step limit of the same idea. It is safe exactly when the answers don’t constrain each other given the input. When they do, answering them in one step has the toy’s problem, and needs either sequential steps or a structure that encodes the constraint.

Where this leads

• World Models, JEPA and Recursive Self-Improvement returns to generative models as world models, which predict what happens next in an environment rather than the next image.

Takeaways

• The forward process adds Gaussian noise in $T$ steps but has a one-jump closed form, $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\varepsilon$; with the linear schedule, signal-to-noise falls to 1 at step 250 of 1,000.
• The ELBO, with the forward posterior in closed form, reduces to a regression: predict the added noise. No adversary, no encoder.
• DDIM reuses a model trained for 1,000 steps to sample in far fewer: on eight modes, 20 steps lost little and 10 about a third of the sharpness. At 5, where the steps fall decides the result: 3% near a mode with even spacing, 39–50% with the DDIM paper’s quadratic spacing.
• Classifier-free guidance mixes conditional and unconditional predictions. Here $w = 1$ always hit the right mode and larger $w$ overshot outward; on images, moderate $w$ helps prompt adherence and high $w$ distorts.
• Latent diffusion runs the process on a VAE’s compressed latents: VAEs and GANs’s VAE as a space, not a sampler.
• Masked diffusion for text trains with a $1/t$-weighted masked-token loss and decodes many tokens per step, independently. When tokens constrain each other, that independence breaks outputs, exactly as computed on a toy language; the speed comes from filling idle arithmetic, so it holds at small batches.

References

Start with Ho et al.’s DDPM paper and DDIM for sampling, then Ho and Salimans for guidance and Rombach et al. for latent diffusion; for text, MDLM and LLaDA.

• Sohl-Dickstein et al., Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015): the original idea of learning to reverse a gradual noising process.
• Ho, Jain and Abbeel, Denoising Diffusion Probabilistic Models (2020): DDPM, the linear schedule used here and the unweighted $\varepsilon$-prediction loss.
• Song and Ermon, Generative Modeling by Estimating Gradients of the Data Distribution (2019): score-based models, the same method reached from the score.
• Song, Meng and Ermon, Denoising Diffusion Implicit Models (ICLR 2021): DDIM’s deterministic, step-skipping sampler and its quadratic spacing.
• Nichol and Dhariwal, Improved Denoising Diffusion Probabilistic Models (2021): the cosine schedule.
• Lipman et al., Flow Matching for Generative Modeling (2022); Liu et al., Flow Straight and Fast: Rectified Flow (2022): the straight-path objectives newer image models train with.
• Song et al., Consistency Models (2023): jumping from any noise level to the clean point in one step.
• Ho and Salimans, Classifier-Free Diffusion Guidance (NeurIPS workshop 2021; arXiv 2022): classifier-free guidance and its $(1 + w)$ convention.
• Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models (2022): latent diffusion, the VAE latent and cross-attention conditioning.
• Peebles and Xie, Scalable Diffusion Models with Transformers (2023): DiT, a Transformer in place of the U-Net.
• Austin et al., Structured Denoising Diffusion Models in Discrete State-Spaces (2021): D3PM, diffusion on discrete tokens, with masking as one corruption.
• Sahoo et al., Simple and Effective Masked Diffusion Language Models (2024): MDLM, the $1/t$-weighted masked-token loss.
• Shi et al., Simplified and Generalized Masked Diffusion for Discrete Data (2024): the same loss, derived independently.
• Nie et al., Large Language Diffusion Models (2025): LLaDA, an 8B masked diffusion model and its semi-autoregressive sampling.
• Arriola et al., Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (ICLR 2025): block diffusion, autoregressive across blocks so finished blocks can be cached.
• Wu et al., Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding (2025): key-value caching for full-sequence diffusion LMs.
• Inception Labs, Mercury: Ultra-Fast Language Models Based on Diffusion (2025): the speed claim for a commercial diffusion LM.