How do diffusion models generate images?

They never generate an image in one shot. Instead they learn to do one small, tractable thing — take a slightly noisy image and predict the noise in it — and then repeat that thousands of times, starting from pure static. Training runs the process backwards from real data: you take a clean image, add a precise, scheduled amount of Gaussian noise to it, and train a network to guess exactly the noise you added. The DDPM paper writes this as the objective of predicting ε, the noise, from the noised image and the timestep. At generation time you start from a canvas of pure random noise and ask the network, step by step, "what noise is in this?" — subtracting a little of its guess each time. After enough steps the static resolves into a coherent image. The genius is that a task no one can do (paint a photo from nothing) is decomposed into a task a network can do very well (denoise a little), performed many times.

What is the forward process and the noise schedule in a diffusion model?

The forward process is the deliberate destruction of an image by adding Gaussian noise in many small steps, and the noise schedule is the recipe that says how much noise each step adds. DDPM defines each step as q(x_t | x_{t-1}) = N(x_t; sqrt(1-β_t) x_{t-1}, β_t I), where β_t is the variance added at step t — small early, larger later. The beautiful part is a closed form: because each step is Gaussian, you can jump straight to any timestep without simulating the ones before it. With ᾱ_t defined as the running product of (1-β_s), the paper gives x_t = sqrt(ᾱ_t) x_0 + sqrt(1-ᾱ_t) ε — the clean image scaled down and a single draw of noise scaled up. DDPM uses T=1000 steps with β increasing linearly from 1e-4 to 0.02, so ᾱ_t glides from nearly 1 (barely touched) down to nearly 0 (pure noise).

Why is training a diffusion model more stable than a GAN?

Because a diffusion model is trained with a plain regression loss against a known target, while a GAN is trained as a two-player game with no fixed target. A GAN pits a generator against a discriminator — Goodfellow's original paper frames it as a "two-player minimax game" — and the two must improve in lockstep; if the discriminator gets too good the generator gets no useful gradient, and generators famously collapse to producing a handful of samples that fool the current discriminator (mode collapse). A diffusion model has no adversary. At every step it is handed a noised image and asked to predict the exact noise that was added — a target you literally have on hand because you added it. That is a simple mean-squared-error objective, optimized like any supervised regression, which is why diffusion training is far more stable and covers the full data distribution instead of a few modes.

What is classifier-free guidance and the guidance scale?

Classifier-free guidance is the dial that trades a text-to-image model's prompt fidelity against its diversity, and it is the single knob users feel most. During training the model is shown the prompt most of the time and, some fraction of the time, nothing at all — Ho and Salimans "jointly train the unconditional and conditional models simply by randomly setting c to the unconditional class identifier ∅ with some probability." At generation you then run the network twice per step, once with the prompt and once without, and push the prediction in the direction the prompt adds: ε̃ = (1+w) ε(z,c) − w ε(z). The guidance scale w is that push. Turn it up and images hew harder to the prompt but grow saturated and samey; turn it down and they get more varied but drift off-prompt. The paper observes "a clear trade-off... with FID monotonically decreasing and IS monotonically increasing with w."

Why does Stable Diffusion run in latent space instead of on pixels?

Because most of a pixel image is perceptually irrelevant detail that is enormously expensive to denoise. The latent-diffusion paper notes that "most bits of a digital image correspond to imperceptible details," yet a pixel-space model still evaluates its network "on all pixels, leading to superfluous computations." The fix is two-stage: first train an autoencoder that compresses a 512×512×3 image down to a small latent grid — at downsampling factor 8 that is 64×64×4 — then run the entire diffusion process in that latent space and decode once at the end. The arithmetic is stark: 512×512×3 is 786,432 values, 64×64×4 is 16,384, exactly 48× fewer, and because attention cost grows with the square of the token count the real saving is larger still. That compression is what moved high-resolution synthesis onto ordinary GPUs.

Diffusion Models, Honestly

You can't paint a photo in one stroke, but you can clean up a slightly-noisy one — and if you can do that, you can generate. That is the whole trick. We destroy a real 2-D distribution on the DDPM noise schedule (the ᾱ_t math, computed live as you scrub time), state the ε-prediction objective in one honest equation, count the network evaluations a 50-step image really costs, work the arithmetic that makes latent diffusion 48× cheaper than pixels, dial classifier-free guidance between fidelity and diversity, and end on the honest history — why the field left GANs behind — and the failure modes nobody screenshots.

Concept · AI / ML. The source ↗

A free, interactive, animated visual explainer of Diffusion Models, Honestly — built to be understood, not skimmed.

Questions

How do diffusion models generate images?: They never generate an image in one shot. Instead they learn to do one small, tractable thing — take a slightly noisy image and predict the noise in it — and then repeat that thousands of times, starting from pure static. Training runs the process backwards from real data: you take a clean image, add a precise, scheduled amount of Gaussian noise to it, and train a network to guess exactly the noise you added. The DDPM paper writes this as the objective of predicting ε, the noise, from the noised image and the timestep. At generation time you start from a canvas of pure random noise and ask the network, step by step, "what noise is in this?" — subtracting a little of its guess each time. After enough steps the static resolves into a coherent image. The genius is that a task no one can do (paint a photo from nothing) is decomposed into a task a network can do very well (denoise a little), performed many times.
What is the forward process and the noise schedule in a diffusion model?: The forward process is the deliberate destruction of an image by adding Gaussian noise in many small steps, and the noise schedule is the recipe that says how much noise each step adds. DDPM defines each step as q(x_t | x_{t-1}) = N(x_t; sqrt(1-β_t) x_{t-1}, β_t I), where β_t is the variance added at step t — small early, larger later. The beautiful part is a closed form: because each step is Gaussian, you can jump straight to any timestep without simulating the ones before it. With ᾱ_t defined as the running product of (1-β_s), the paper gives x_t = sqrt(ᾱ_t) x_0 + sqrt(1-ᾱ_t) ε — the clean image scaled down and a single draw of noise scaled up. DDPM uses T=1000 steps with β increasing linearly from 1e-4 to 0.02, so ᾱ_t glides from nearly 1 (barely touched) down to nearly 0 (pure noise).
Why is training a diffusion model more stable than a GAN?: Because a diffusion model is trained with a plain regression loss against a known target, while a GAN is trained as a two-player game with no fixed target. A GAN pits a generator against a discriminator — Goodfellow's original paper frames it as a "two-player minimax game" — and the two must improve in lockstep; if the discriminator gets too good the generator gets no useful gradient, and generators famously collapse to producing a handful of samples that fool the current discriminator (mode collapse). A diffusion model has no adversary. At every step it is handed a noised image and asked to predict the exact noise that was added — a target you literally have on hand because you added it. That is a simple mean-squared-error objective, optimized like any supervised regression, which is why diffusion training is far more stable and covers the full data distribution instead of a few modes.
What is classifier-free guidance and the guidance scale?: Classifier-free guidance is the dial that trades a text-to-image model's prompt fidelity against its diversity, and it is the single knob users feel most. During training the model is shown the prompt most of the time and, some fraction of the time, nothing at all — Ho and Salimans "jointly train the unconditional and conditional models simply by randomly setting c to the unconditional class identifier ∅ with some probability." At generation you then run the network twice per step, once with the prompt and once without, and push the prediction in the direction the prompt adds: ε̃ = (1+w) ε(z,c) − w ε(z). The guidance scale w is that push. Turn it up and images hew harder to the prompt but grow saturated and samey; turn it down and they get more varied but drift off-prompt. The paper observes "a clear trade-off... with FID monotonically decreasing and IS monotonically increasing with w."
Why does Stable Diffusion run in latent space instead of on pixels?: Because most of a pixel image is perceptually irrelevant detail that is enormously expensive to denoise. The latent-diffusion paper notes that "most bits of a digital image correspond to imperceptible details," yet a pixel-space model still evaluates its network "on all pixels, leading to superfluous computations." The fix is two-stage: first train an autoencoder that compresses a 512×512×3 image down to a small latent grid — at downsampling factor 8 that is 64×64×4 — then run the entire diffusion process in that latent space and decode once at the end. The arithmetic is stark: 512×512×3 is 786,432 values, 64×64×4 is 16,384, exactly 48× fewer, and because attention cost grows with the square of the token count the real saving is larger still. That compression is what moved high-resolution synthesis onto ordinary GPUs.

Diffusion Models, Honestly

Questions

Related explainers