12.4 Synthetic data and distillation at scale

What this is and why it exists

Hand-labelling is the bottleneck in almost every applied project, and generating training data with a strong model removes it — which is what much of the recent small-model progress actually runs on. The technique is straightforward and the leverage is entirely in the filtering, which is the step people skip. Skipping it invites two problems worth understanding before you build: a narrowing of what your model can produce, and a licence question about the model you generated from.

The vocabulary

• Synthetic data — training examples produced by a model rather than collected.
• Teacher and student — the strong model generating, and the small model trained on it.
• Distillation — training the student to reproduce the teacher's behaviour.
• Filtering — discarding generated examples that fail a check.
• Diversity — how much variety a dataset contains.
• Model collapse — the narrowing that comes from training repeatedly on generated data.
• Provenance — the record of where each example came from.

The mental model

The basic move is three steps and the middle one is where the value is. Generate candidate examples with a strong model. Filter them hard. Train a small model on what survives.

Filtering matters more than generation because a strong model produces plenty of output that is wrong, repetitive, trivial, or subtly off in exactly the way that teaches a student a bad habit. Training on unfiltered generation transfers the teacher's mistakes as ground truth, and the student learns them with confidence.

The filters that work, in roughly increasing order of value:

Deterministic checks, where they exist and they exist more often than people look: does the code run, do the tests pass, does the arithmetic evaluate, does the output parse against the schema, is the extracted value actually present in the source text. These are the strongest filters available and they cost nothing — a generated example that fails a real check is discarded with certainty rather than judgement.

Consistency: generate the same item several times and keep it only where the attempts agree. Disagreement is a reliable signal that the teacher was unsure, and unsure output is exactly what you do not want as a label.

Deduplication, near-duplicate as well as exact, because generation repeats itself far more than it appears to and a dataset of a hundred thousand examples can contain a few thousand distinct ideas.

Difficulty: discard the trivial cases, which teach nothing, and keep the hard cases where the student needs the teacher most.

And human review of a sample, always, before trusting the set — a few hundred examples read by a person, because that is what catches the systematic error a filter was not designed for.

Diversity has to be engineered rather than hoped for, and this is the second thing people underestimate. Asked repeatedly for examples, a model produces variations on a narrow theme. Vary it deliberately: seed each generation from real data — a real document, a real question, a real customer segment — vary the requested style, length, difficulty and perspective explicitly, sample at a higher temperature for generation than you would for answering, and measure the spread of what you got rather than assuming it. Clustering the generated set and looking at how many clusters there are is a five-minute check that regularly surprises people.

Teacher-student distillation is the most reliable technique in the topic, and it is the one to reach for. Use the strong model to produce labelled examples for your specific task, train a small model on them, and serve the small one. The student does not acquire the teacher's general ability and does not need to; it acquires the narrow competence you trained for, at a fraction of the running cost and latency. This is the hybrid the encoder topic and the compression topic both recommended, and it is the most consistently valuable move in applied work with these systems — the expensive model is paid for once during development rather than on every request forever.

Model collapse is the failure that makes synthetic data not free. Train on generated data, generate from that model, train again — and the variety narrows at each round. The tails of the distribution go first: rare phrasings, unusual cases, minority patterns, the long tail that made the original data valuable. Each generation reproduces the centre of what it saw and drops the edges, and after a few rounds the output is fluent, plausible and much narrower than where it started.

Two practical consequences. Do not train recursively on your own generations without fresh real data, which is the direct form of the problem. And be aware of the indirect form: models increasingly train on text scraped from a web that now contains a great deal of generated text, so the effect operates without anybody choosing it, which is a live concern for the field rather than for your project.

The defences for your own work are ordinary. Keep real data in the mixture rather than replacing it. Measure diversity across rounds instead of assuming it holds. Anchor generation to real seeds. And treat synthetic data as an amplifier of what you have rather than a substitute for having any.

Then the licensing and provenance questions, which are unsettled and worth knowing exist before you build on them.

The teacher's terms may restrict this. Several providers' terms prohibit using their outputs to train a competing model, and the precise scope of that — what counts as competing, whether a narrow task model counts — is exactly the kind of question that has not been settled. Read the terms of the model you generate from, before generating, not after training. For open-weight models, the licence still applies and some carry restrictions of their own.

The data behind the teacher raises its own question, unresolved in most jurisdictions: whether outputs derived from a model trained on copyrighted material carry any obligation. Nobody sensible is certain, and it is not resolved by anyone's confidence on the internet.

What you can do is keep provenance, which is both good practice and the thing that makes a future obligation actionable rather than catastrophic. Record for every example: which model produced it, when, from what prompt and what seed data, which filters it passed, and whether a person reviewed it. If a term changes, or a set turns out to be tainted, you can identify and remove exactly what came from that source — and without the record you would have to discard everything.

What you should now be able to explain or do

State the three steps and say why filtering carries the value. Apply deterministic, consistency, deduplication and difficulty filters, plus human review of a sample. Engineer diversity with real seeds and explicit variation, and measure the spread. Run teacher-student distillation for a narrow task and say why the student needs no general ability. Explain model collapse, which part of the distribution goes first, and both the direct and indirect forms. Apply the four defences. Read the teacher's terms before generating, and keep provenance that makes a later problem removable.

Check yourself