12.4 Synthetic data and distillation at scale
You can build a training set without labelling by hand.
Before:08. Large Language Models
Synthetic data and distillation manufacture training sets without hand-labelling: generate with a strong model, filter hard, train the student — and much of the current small-model boom runs on exactly this. It sits in the frontier module as the data-side frontier. The leverage lives in the filter, and skipping it is the trap; recursive training on unfiltered generations degrades quality generation over generation, a decay with its own name — model collapse.
Work through these
Generating and filtering synthetic data
Generating training data with a model, then filtering it to keep what is actually useful. The filtering matters more than the generation.
Teacher–student distillation
Training a smaller model on the outputs of a larger one, which transfers much of the capability at a fraction of the running cost. It is the most reliable technique in this topic.
Model collapse and diversity loss
Training repeatedly on generated data narrows the variety of what a model produces. Understanding this failure is what stops synthetic data being treated as free.
Licensing and provenance of generated data
Generated data carries obligations from the model that produced it and from the data behind that model. These are unsettled questions worth knowing exist before building on them.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.