Generative Models

01Intuition

Generative vs. discriminative: what separates them

A classifier that tells cats from dogs only needs to find a boundary between the two classes. It never has to understand what makes a photo look like a plausible photo at all.

That is a discriminative model. It learns $P(\text{label}\mid\text{data})$, a function that maps inputs to outputs and nothing more.

A generative model is asked to do something much harder. It learns $P(\text{data})$ itself: the full distribution of "things that look like real photos". Well enough that you can draw a fresh sample from it and get something that was never in the training set but still looks like it belongs there.

Discriminative modeling asks for a boundary. Generative modeling asks you to understand the shape of the whole space well enough to invent a new point inside it.

Autoregressive generation — the idea from page 10, generalized

The simplest way to build a generative model is the one you already know from language models (10). Factor the joint distribution into a product of conditionals and predict one piece at a time: $P(x) = \prod_i P(x_i \mid x_{.

For text that is next-token prediction. For images the same idea applies pixel by pixel, or more practically over discrete tokens produced by compressing the image first. Generate the top-left pixel, then the next one conditioned on it, and so on.

It works, and it gives an exact likelihood. Two things hold it back:

• Speed. Thousands of sequential steps for a single image.
• No revision. Once a pixel or token is generated, it is fixed. There is no notion of going back to improve an earlier guess.

That second limitation is a big part of why the field went looking for other factorizations of the generation process.

VAEs: compress, then regularize the compression

A plain autoencoder learns two things: an encoder that compresses an input into a low-dimensional latent code, and a decoder that reconstructs the input from that code. Train it well and reconstruction error goes to nearly zero.

But a plain autoencoder is not a generative model. Its latent space is just whatever shape minimizes reconstruction loss, with no constraint that nearby points decode to anything sensible. A random sample from it lands where the decoder has seen nothing like it, and you get garbage.

A Variational Autoencoder fixes exactly this, with two changes:

• The encoder no longer outputs a single latent code. It outputs the parameters of a distribution (a mean and a variance) over latent codes, and a sample is drawn from that distribution before being handed to the decoder.
• The loss adds a KL-divergence term that pulls every input's latent distribution toward a fixed, simple prior, usually a standard Gaussian.

import torch
from torch.distributions import Normal
r = lambda t: [round(v, 3) for v in t.tolist()]

mu = torch.tensor([0.5, -1.0], requires_grad=True)
sd = torch.tensor([0.8, 0.3], requires_grad=True)
q = Normal(mu, sd)            # what the encoder emits for one input

torch.manual_seed(0)
z_bad = q.sample()            # the draw happens inside the graph
print("sample() ", r(z_bad), "grad_fn", z_bad.grad_fn)
try:
    z_bad.sum().backward()
except RuntimeError as e:
    print("backward:", str(e).split(" and")[0])

torch.manual_seed(0)
z_ok = q.rsample()            # the randomness is pushed out to eps
print("rsample()", r(z_ok), "grad_fn", type(z_ok.grad_fn).__name__)
z_ok.sum().backward()
print("dL/dmu   ", r(mu.grad), " dL/dsd", r(sd.grad))

torch.manual_seed(0)
eps = torch.randn(2)          # the same draw, written out by hand
print("by hand  ", r(mu + sd * eps), " eps   ", r(eps))

GANs: two networks playing a game

A GAN trains two networks against each other:

• The generator maps random noise to a fake sample, trying to fool the other network.
• The discriminator looks at real and fake samples and tries to tell them apart.

Train both simultaneously and, in the ideal case, the generator gets pushed to produce samples indistinguishable from real data. No explicit density is ever modeled. The generator just learns to sample well.

GANs have two genuine, well-documented failure modes that were never fully solved in general, and they are not minor historical footnotes:

• Mode collapse. The generator discovers a handful of outputs that reliably fool the current discriminator and stops producing anything else. Diversity collapses even while individual samples look fine.
• Training instability. Two networks chase a moving target with no natural stopping criterion. The pair is prone to oscillation, and vulnerable to a discriminator that gets too good too fast and leaves the generator with a vanishing gradient signal.

Careful architectures and loss variants (Wasserstein GANs, spectral normalization, and others) reduced how often this happens. But "will this GAN converge nicely" was never something you could assume.

Diffusion: learn to un-shred a photo, one small step at a time

The idea that made everything above easier to avoid: never ask a network to go straight from noise to a perfect image in one shot.

Take a real image and destroy it. Add a small amount of Gaussian noise, then a bit more, then a bit more, hundreds of times, until what is left is indistinguishable from pure static. This is the forward process, and nothing in it is learned: it is a fixed, known mathematical recipe.

Now train a network to do the single easiest version of the reverse job. Given a slightly-too-noisy image, predict what noise was added and remove a little of it. Not "generate the whole image". Just "make this one step less noisy".

Chain that one easy step backward, from pure noise all the way to $t{=}0$, and you have generated a new image. One small, well-defined denoising decision at a time.

That framing is what buys diffusion both of its advantages:

• More stable than a GAN. At every step there is a well-defined right answer (the noise that was added), so the network is doing ordinary supervised regression rather than chasing an adversary.
• Sharper than a plain VAE. A VAE asks one decoder to nail the entire reconstruction in a single forward pass, which encourages hedging toward a blurry average. Diffusion spreads the work across hundreds of small, low-stakes corrections, each of which the network can learn to do well.

02Timeline

03Architecture: how the data flows

Two processes run in opposite directions over the same sequence of increasingly noisy images $x_0, x_1, \dots, x_T$:

• The forward process $q$ is fixed and requires no learning. It is just "add a bit of Gaussian noise, according to a known schedule".
• The reverse process $p_\theta$ is the one neural network in this whole picture. It is trained to undo one forward step at a time.

Reference figureThe forward (reverse) diffusion Markov chain, as drawn in the DDPM paper. Source: Ho et al. 2020, via Lilian Weng, What are Diffusion Models?

What does $\epsilon_\theta$ look like as a network? Two answers, a few years apart:

• U-Net, originally. A convolutional encoder-decoder with skip connections between matching resolutions, the same backbone family used in image segmentation. A natural fit, because input and output are both full images: a noisy image in, a predicted noise map out.
• Diffusion Transformer (DiT), in more recent large-scale systems. Chop the (usually latent-space) image into patches, treat them like tokens, and run ordinary transformer blocks. The same architecture from 08, applied to image patches instead of text tokens, with cross-attention or embedded conditioning bringing in the prompt at every block.

DiT tends to scale more predictably with more compute and data, the same lesson the field already learned from the original NLP transformers.

Conditioning gets folded in at every denoising step. It might be text, a pose skeleton, a depth map, or an edge map. Two common mechanisms: cross-attention (query from the image, key/value from the conditioning), or directly adding a conditioning embedding into the network.

ControlNet-style approaches take this further. Freeze a pretrained diffusion model entirely, then train a small parallel branch that reads an extra conditioning signal (edges, pose, depth) and injects its influence into the frozen network's intermediate activations. That gives precise structural control, exact pose and exact layout, without retraining or damaging the original model's learned knowledge.

04The equations

The forward process is designed so you can jump straight to any noise level $t$ in one step, without simulating all the steps before it: $x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1-\bar\alpha_t}\, \epsilon$.

Here $\epsilon \sim \mathcal{N}(0, I)$ is fresh Gaussian noise, and $\bar\alpha_t$ is a known, fixed schedule value that decreases from near 1 (almost no noise) to near 0 (almost pure noise) as $t$ increases. That is the whole forward process, with no network involved and only algebra.

import numpy as np
rng = np.random.default_rng(0)

T = 1000
betas = np.linspace(1e-4, 0.02, T)      # the DDPM linear schedule
abar = np.cumprod(1.0 - betas)          # alpha-bar_t, one per step
x0 = np.array([0.9, -0.7, 0.5, -0.3])   # a 4-pixel "image"

# The slow way: 400 real noising steps, on 20k copies at once.
x = np.repeat(x0[None], 20000, axis=0)
for t in range(400):
    z = rng.normal(size=x.shape)
    x = np.sqrt(1 - betas[t]) * x + np.sqrt(betas[t]) * z
print("x", tuple(x.shape))
print("400 steps:", np.round(x.mean(0), 3),
      "+/-", round(float(x.std(0).mean()), 3))

a = abar[399]                           # the same law, no loop at all
print("1 step:   ", np.round(np.sqrt(a) * x0, 3),
      "+/-", round(float(np.sqrt(1 - a)), 3))

for t in (0, 250, 500, 999):
    xt = np.sqrt(abar[t]) * x0 + np.sqrt(1 - abar[t]) * rng.normal(size=4)
    print(f"t={t:4d} sqrt(abar)={np.sqrt(abar[t]):.3f}",
          "x_t=", np.round(xt, 2))

Before the diffusion loss, the VAE's. Diffusion inherits its whole argument, so it is worth having this one straight first.

import torch

x = torch.tensor(2.0)      # one data point; the decoder is the identity

def terms(mu, logsd):
    sd = logsd.exp()
    recon = (mu - x) ** 2 + sd ** 2          # E_q[(x - z)^2], exactly
    kl = 0.5 * (mu ** 2 + sd ** 2 - 1) - logsd
    return recon, kl

print(f"{'beta':>5}{'mu':>8}{'sigma':>8}{'recon':>8}{'KL':>8}")
for beta in [0.01, 0.1, 1.0, 4.0, 50.0]:
    p = torch.zeros(2, requires_grad=True)   # [mu, log sigma]
    opt = torch.optim.Adam([p], lr=0.02)
    for _ in range(4000):                    # this is all training does
        recon, kl = terms(p[0], p[1])
        opt.zero_grad()
        (recon + beta * kl).backward()       # the only knob is beta
        opt.step()
    recon, kl = terms(p[0], p[1])
    print(f"{beta:5.2f}{p[0]:8.3f}{p[1].exp():8.3f}"
          f"{recon:8.3f}{kl:8.3f}")

Training the reverse process reduces, after the derivation settles, to a strikingly simple objective:

• x_0 a real training image, sampled from the dataset.
• t a timestep sampled uniformly at random from $1$ to $T$ — every training step picks a different, random noise level to train on.
• ε the actual Gaussian noise you drew and added to $x_0$ to construct $x_t$ via the formula above — this is the ground-truth label.
• x_t the resulting noisy image at that timestep — this is what the network sees as input, along with $t$ itself.
• ε_θ(x_t, t) the network's prediction: "given this noisy image and this timestep, what noise do I think was added?"
• ‖·‖² plain L2 distance between the true noise and the predicted noise — an ordinary regression loss, nothing adversarial or exotic about it.

Notice what's not in this loss: no discriminator, no generator fighting anything, no minimax. Pick a random image, pick a random timestep, add the corresponding amount of noise, ask the network to predict that noise, compute MSE, backpropagate. It's supervised regression with an unusually clever choice of label.

At sampling time you start from $x_T \sim \mathcal{N}(0, I)$, pure noise. Then repeatedly use $\epsilon_\theta(x_t, t)$ to estimate and subtract off a bit of noise, stepping from $x_T$ down to $x_{T-1}$, then $x_{T-2}$, and so on to $x_0$.

The original DDPM sampler does this stochastically, adding a little fresh noise back in at each step to match the forward process exactly. It needs on the order of hundreds to a thousand steps.

DDIM reformulates the reverse process as non-Markovian and deterministic. Same trained network, but a sampling procedure that can skip steps and still land close to a good sample. That cuts generation from hundreds of steps to a few dozen with the same weights, which is the difference between diffusion being usable in a real product and not.

05Why this architecture

Why diffusion trains more stably than a GAN. A GAN's loss is defined by two networks simultaneously trying to outmaneuver each other, so there is no fixed target: the "correct" gradient for the generator changes every time the discriminator updates, and nothing guarantees the pair converges rather than oscillates.

Diffusion training has none of that. At every step there is a single, fixed, correct answer (the noise that was added), so it is ordinary supervised regression. No adversary means no oscillation and no mode collapse in the GAN sense. It also means a loss curve that corresponds to the model getting better, which makes debugging and hyperparameter tuning tractable in a way GAN training famously isn't.

Why diffusion runs in latent space. Running the forward/reverse process directly on pixels means every one of those hundreds of denoising steps is a full forward pass of a large network over a full-resolution image. At 512×512×3 or larger, that is expensive to the point of being impractical on consumer hardware.

Latent diffusion first trains a VAE-style encoder/decoder pair that compresses images into a much smaller latent grid. Commonly an 8× spatial downsampling, so a 512×512 image becomes something like a 64×64 latent. The entire diffusion process then runs inside that compressed space: forward noising, reverse denoising, all of it. The final denoised latent is decoded back to pixels with the VAE decoder only once, at the very end.

It is the single change that took diffusion from a research curiosity requiring serious compute to something that runs on a single consumer GPU, which is exactly why it is called Stable Diffusion.

06Complexity, failure modes, and what breaks

What breaks

• Mode collapse (GAN). The generator finds a small set of outputs that reliably fool the current discriminator and stops exploring the rest of the data distribution. You get variety-starved samples: individually convincing, collectively failing to cover the real diversity of the training data. Mitigations (Wasserstein loss, minibatch discrimination, spectral normalization) reduce how often this happens. None eliminate it as a possibility.
• Blurry samples (VAE). A pixel-wise reconstruction loss, averaged over every plausible way an ambiguous region could look, is minimized by outputting something close to the average of those possibilities. That average looks blurry. Stronger decoders and adversarial or perceptual losses on top of the VAE objective help. But the core tension never fully disappears: KL regularization is needed for a samplable latent space, and it pulls against sharp reconstruction.
• Slow multi-step sampling (diffusion). The original DDPM sampler needs hundreds to a thousand sequential network evaluations per sample. Each one is a full forward pass, and they cannot be parallelized, because each step depends on the output of the last. DDIM is the direct first response: deterministic, skip steps, same weights. Further speedups come from distillation techniques that train a student model to match what the multi-step teacher would have produced in far fewer steps, sometimes down to single digits.

Two other model families deserve a mention, even though they get less production use today.

• Normalizing flows. Build a generative model out of a sequence of invertible transformations. That gives you an exact log-likelihood via the change-of-variables formula, a real advantage over the approximate bounds every other family here relies on. The cost: every layer has to be restricted to invertible, easy-to-Jacobian-determinant operations. That constrains architecture choices, and has kept flows from matching GAN or diffusion sample quality at scale.
• Energy-based models. Instead of directly defining how to sample, define a scalar "energy" function that should be low for realistic data and high for everything else. Generation means searching for low-energy points, typically via iterative gradient-based sampling such as Langevin dynamics. Conceptually elegant, and closely related to score-based views of diffusion. But training them (typically via contrastive divergence) and sampling from them efficiently is hard, which has limited their production use relative to diffusion.

Evaluating generative models

Unlike a classifier, there is no single clean accuracy number here. Two imperfect options:

• FID (Fréchet Inception Distance) compares the statistics of real and generated image features under a pretrained classifier network. It is the most commonly reported number in papers. It is also a proxy: sensitive to the reference dataset and the feature extractor used, and known to not always track what humans prefer.
• Human preference evaluation (pairwise comparisons, "which image do you prefer") is closer to what matters for a product. It is expensive, slow, and still noisy across raters.

Use both as proxies to triangulate with. Neither is ground truth: a model can win on FID and still look worse to people, or the reverse.

07Build this

Classifier-free guidance

This is the single most practically important diffusion concept if you are using these models, and it is simpler than it sounds.

During training, the conditioning signal (a text prompt's embedding, say) is randomly dropped and replaced with a fixed "empty" token some fraction of the time. A common choice is around 10%. So the exact same network learns to predict noise two ways:

• $\epsilon_\theta(x_t, t, c)$: the conditional prediction, given the prompt.
• $\epsilon_\theta(x_t, t, \varnothing)$: the unconditional prediction, ignoring the prompt entirely.

At sampling time, compute both predictions and extrapolate away from the unconditional one, toward the conditional one, by more than the conditional prediction alone would give you:

where $w$ is the guidance scale. Illustrative: at $w{=}1$ you get the plain conditional prediction, with no extrapolation.

Push $w$ higher and you exaggerate the direction the prompt is already pulling the prediction, which produces images that adhere to the prompt much more strongly. Values in roughly the 7–15 range are commonly used defaults in practice; Stable Diffusion's original default sat around 7.5.

Push it too far and you trade away diversity, and start seeing oversaturated colors and artifacts. Guidance scale is a real dial with a real quality/adherence tradeoff, and there is no free lunch.

Guidance is the dial. The reverse process is the thing being steered, and it is small enough to build from scratch in an afternoon.

Where this runs in production

Three distinct pretrained components, chained together, each doing one well-scoped job:

1. Text encoder. A pretrained language model or CLIP-style text tower turns the prompt string into a conditioning embedding.
2. Latent diffusion U-Net or DiT. Starts from random noise in latent space and runs the reverse denoising loop, typically a few dozen steps under DDIM-style sampling, with classifier-free guidance applied at every step to sharpen prompt adherence.
3. VAE decoder. The final denoised latent passes through once, producing the actual pixel-space image.

Guidance is not a fourth model. It carries no weights of its own: it runs the same denoising network twice per step, once conditioned and once not, then extrapolates between the two predictions.

Video generation runs the same core machinery, but adds a new problem: temporal consistency. It is not enough for every individual frame to look right. Consecutive frames need to agree on where objects are, how they are moving, and what they look like. Otherwise you get flicker, and objects that subtly warp between frames.

There are two approaches. One extends the denoising network with temporal attention or convolution layers that let it look across nearby frames. The other diffuses over a whole short clip's latent representation jointly, instead of generating each frame's noise independently. Both treat time as another axis the network has to get right and not as an afterthought bolted onto single-image diffusion.

08Where you meet this in the wild

Generative modelling stopped being about pretty pictures somewhere around 2022. The interesting uses now are the ones where the thing being sampled is not an image at all.

The pattern to notice: diffusion works wherever you can define a sensible way to add noise and a network that can undo it. Pixels were just the first domain where that was easy. Atoms, weather states, and robot trajectories turned out to work too.

09Going deeper: the decisions that change your output

Everything above explains why diffusion works. This section is about the handful of choices that decide what you get out of it.

Inside Stable Diffusion, at inference

The pipeline above listed three components, but it did not say what each one costs you at generation time, and the distribution is lopsided.

• The text encoder runs once and is cheap enough to ignore.
• The denoising network runs once per step, and twice per step when guidance is on, so it is the entire compute budget.
• The VAE decoder runs once, at the end. Also cheap next to the loop.

So a 30-step generation with classifier-free guidance is 60 forward passes of a large network. Everything else rounds to zero.

The knobs, roughly in order of how much they move the result:

• Step count. Cost is linear in it. How low you can go depends entirely on the sampler.
• Guidance scale. Prompt adherence traded against diversity, covered above.
• Sampler. Changes the step floor by an order of magnitude, and costs nothing to swap.
• Resolution. Cost is quadratic in it, and quality also tends to degrade well away from the resolution the model was trained at.

Samplers: same weights, different arithmetic

A sampler is the rule for stepping from $x_t$ to $x_{t-1}$, and it is not part of the trained model at all. Swap it whenever you like.

DPM-Solver treats the reverse process as an ODE and applies a higher-order numerical solver to it. Same network, same weights, better integration. Its authors report high-quality samples in 10 to 20 network evaluations, a 4 to 16× speedup over previous training-free samplers.

Adaptation without retraining

Almost nobody trains one of these from scratch, so practical work is nearly all adaptation of a checkpoint someone else paid for. Five techniques cover most of what people do, and they differ mainly in how much of the model they are willing to disturb.

• LoRA. Freeze the base weights and learn a small low-rank update beside them. Proposed for language models, it transfers directly to diffusion. The files are small enough to swap and stack freely.
• DreamBooth. Fine-tune the model on a handful of images of one subject, bound to a rare identifier token. Strong subject fidelity. Also the heaviest option, and the most prone to overfitting a small reference set.
• Textual inversion. Keep the model entirely frozen and learn one new embedding vector for one new token. The smallest possible intervention, and correspondingly limited in what it can express.
• ControlNet. Clone the encoder and train the copy on a spatial conditioning signal: edges, depth, pose, segmentation. It controls composition, with content left to the prompt.
• IP-Adapter. Add a separate cross-attention path for image features, so an image can act as a prompt alongside the text. Roughly 22M parameters, and it composes with the others.

ControlNet has a detail worth stealing for your own work. The trained copy reconnects to the frozen base through zero-initialised convolutions. At the start of training those layers output nothing, so the branch is a no-op and cannot damage a working model. It grows its influence from zero as training proceeds.

Where GANs still win

Not many places, but the ones that remain are real and worth being precise about.

A GAN generator produces its output in one forward pass. Diffusion needs several at absolute minimum, usually dozens. When the latency budget is small and fixed, that gap decides the architecture on its own.

• Neural vocoding. HiFi-GAN emits a waveform in a single pass. A voice agent replying in real time cannot afford a denoising loop. See 13.
• Super-resolution and restoration. Real-ESRGAN-style models are still widely deployed for upscaling, where one pass is enough.

Notice what those share: the output is tightly constrained by the input, so the one-to-many problem that drives mode collapse is much smaller to begin with. Add a hard latency budget and the trade tips toward the GAN.

Few-step generation

Better samplers took diffusion from a thousand steps down to roughly ten, which is where training-free cleverness runs out. Going lower means changing the weights as well as integrating them more carefully.

Consistency models are trained so that every point along a trajectory maps directly to that trajectory's endpoint. If that property holds, you can jump straight to the answer instead of walking to it.

Latent consistency models apply the idea to an existing latent diffusion model by distilling it. The authors report 2 to 4 step generation at 768×768, distilled in roughly 32 A100-hours.

The trade is consistent across the whole family. You give up some quality and some diversity, you gain a large cut in latency, and you pay a distillation run up front.

Evaluation is the bottleneck

The failure-modes section introduced FID and human preference. Both are weaker than their ubiquity suggests.

The Rethinking FID analysis lays out the specific problems:

• Inception features were trained for classification. They represent the content of modern text-to-image output poorly.
• FID compares only the first two moments, which assumes the feature distributions are Gaussian, and typically they are not.
• Sample complexity is poor, so the number you get shifts with how many samples you scored.
• It can contradict human raters, and does not reliably track gradual improvement in a model.

That paper proposes CMMD instead, pairing CLIP embeddings with a maximum mean discrepancy distance.

Generative quality is easy to see and hard to measure, and no automated metric settles the question today.

10Interview questions

11Go deeper

●Now write it yourself

Reading the derivation and being able to produce it are different skills. These are Deep-ML problems that exercise what this page covers — each one is checked against real test cases, not multiple choice.

Matched to this page from Deep-ML's catalogue of 1,380 problems. More at deep-ml.com, and Where to practise covers the other platforms and what each one trains.

12Related pages

Other techniques for this problem

A scoped slice of the full Technique Map — every technique this page covers, grouped by what it solves.