6.19 Generative models: GANs and diffusion

What this is and why it exists

Generative image models went from a curiosity to a tool most people have used, and the shift from one dominant approach to another happened for a reason worth understanding: the older objective is structurally difficult to train and the newer one is not. This topic is both, end to end — enough that you can explain how an image generator works from noise to picture, and enough that the practical levers make sense rather than being settings you copy.

The vocabulary

• Generator — a network producing samples from random input.
• Discriminator — a network judging whether a sample is real or generated.
• Mode collapse — a generator producing a few outputs regardless of its input.
• Forward process — gradually adding noise to data until nothing is left.
• Reverse process — the learned removal of that noise, one step at a time.
• Sampler — the procedure that runs the reverse process to produce an image.
• Guidance — dialling up how strongly the output follows its conditioning.
• Latent diffusion — running the whole process in a compressed space rather than on pixels.

The mental model

The adversarial idea is elegant and its difficulties are structural. A generator turns random input into an image; a discriminator judges real against generated; each trains against the other, and the generator improves by learning to fool a judge that is itself improving. The appeal is that nobody has to say what makes an image good — the discriminator learns the criterion.

The difficulties follow from the same design. Mode collapse: if the generator finds one output the discriminator currently accepts, producing only that output is a winning move, so variety disappears while the loss looks fine. Instability: this is two networks in a moving equilibrium rather than one function being minimised, so the loss curves do not indicate progress the way an ordinary loss does — they can look terrible while samples improve and look fine while samples degrade. And balance: a discriminator that wins too easily gives the generator no usable gradient, while one that is too weak provides no signal. Numerous techniques exist to manage all this, and none removes the fact that you are balancing a system rather than descending a surface. That is much of why the field moved on.

Diffusion inverts the problem, and the inversion is what makes it stable. Take an image and add a small amount of noise; repeat until nothing recognisable is left. That is the forward process, and it needs no learning — it is a fixed procedure you define. Now train a network to undo one step: given a noisy image and how far along the process it is, predict the noise that was added. That is an ordinary supervised regression problem with an exact target, because you added the noise yourself and know precisely what it was. There is no adversary, no equilibrium, and the loss means what a loss normally means.

To generate, start from pure noise and run the learned reversal repeatedly, removing a little at each step, and an image emerges. The strikingly different feature of this approach is that generation is an iterative process rather than a single forward pass — which is the source of both its quality and its cost.

The three practical pieces follow from that. The standard training formulation is the one described above: sample a data point, sample how far along to go, add the corresponding noise, and train the network to predict it. Sampling variants address the cost. The original procedure needs many steps, each a full network evaluation, which is slow; the deterministic variants take much larger steps along the same learned trajectory and produce comparable images in a fraction of the number, and being deterministic they also make the same starting noise reproduce the same image, which matters for iterating on a result.

Guidance is the lever that made text conditioning practical. Train the model to work both with the conditioning and without it — sometimes dropping the text during training so it learns both behaviours. At generation time, compute both predictions and extrapolate away from the unconditioned one, which amplifies whatever the conditioning contributed. A dial controls how far. Turn it up and the image follows the prompt more literally, at the cost of variety and, past a point, of realism; turn it down and you get more diverse and less obedient results. It is the single setting that most changes the character of the output, and knowing that it is an extrapolation rather than a strength parameter explains why extreme values produce over-saturated, strangely rigid images.

Latent diffusion is what made all of this affordable. Running hundreds of denoising steps on full-resolution pixels is enormously expensive, and most of those pixels are perceptually redundant. So train an autoencoder — the previous topic's machinery — to compress images into a much smaller representation and back, then run the entire diffusion process in that compressed space, decoding only at the end. The computation drops by a large factor, and the quality holds because the autoencoder preserves what matters perceptually. Text conditioning enters through cross-attention: the text is encoded, and the denoising network attends to it at every step, so the prompt steers every stage of the reversal rather than only the start. That combination — compressed space, iterative denoising, text through cross-attention, guidance as the dial — is the architecture behind the current generation of image tools, and you can now describe it end to end.

Two things to carry beyond the mechanics. Evaluating generative models is genuinely unsettled: the automatic scores in common use measure whether the distribution of outputs resembles the distribution of real images, which is not the same as whether any individual image is good, and human judgement remains the standard for quality. And these models reproduce what they were trained on, including its biases and sometimes its specific content, so the questions of consent, attribution and misuse are part of the engineering rather than a separate ethical appendix.

What you should now be able to explain or do

Describe the adversarial setup and name its three structural difficulties. Explain why adversarial loss curves do not indicate progress. Describe the forward and reverse processes and say why the training objective is an ordinary supervised one. Explain why generation is iterative and what that costs. Say what the faster deterministic samplers change and what determinism buys. Explain guidance as an extrapolation and predict what extreme values do. Describe latent diffusion and how text enters. State why evaluation is unsettled.

Check yourself