12.7 Evaluation science

What this is and why it exists

Public benchmark numbers are quoted constantly and are worth much less than their precision suggests, for reasons that are structural rather than dishonest. Test material leaks into training sets. Popular benchmarks stop discriminating. Small differences on small test sets are noise. This topic is how to read such a number, and the conclusion it drives at: an internal benchmark you control is the only one whose result reliably predicts your outcome.

The vocabulary

• Contamination — test material present in a model's training data.
• Saturation — a benchmark on which everything scores near the top.
• Held-out benchmark — one whose test set is kept private.
• Dynamic benchmark — one whose contents change over time.
• Sampling noise — variation from which examples happened to be in the test set.
• Confidence interval — the range a measured value is plausibly within.
• Overlapping intervals — a sign two results may not differ at all.

The mental model

Contamination is the first problem and it affects nearly every published comparison. Models are trained on enormous scrapes of the web, and benchmarks live on the web — in repositories, papers, blog posts, discussion threads, and tutorials that quote the questions. So the test set is frequently in the training data, and a model that has seen the answers scores well without any of the ability the benchmark was measuring.

It is usually not deliberate, and that does not help. Nobody set out to include it; the filtering did not catch it; a paraphrase of the test set was in a tutorial. And it is genuinely hard to check from outside, since you cannot inspect a training corpus you do not have.

The signals that suggest it are worth knowing. A model that performs far better on a well-known benchmark than on a fresh set of the same kind of question. Performance on old items much higher than on recent ones. Recall of a benchmark's exact phrasing when prompted with a fragment. And the practical rule: treat a benchmark released before a model's training cut-off as potentially contaminated, and treat anything released afterwards as more informative — which is why dated benchmarks and recently constructed ones carry more weight.

Saturation is the second and it is quieter. A benchmark that everything now scores near the top of has stopped discriminating: the remaining differences are within its own label noise, and a comparison between two models on it says nothing. Every widely used benchmark eventually reaches this, because the field optimises against what it measures — which is not cheating, it is what a target does.

Two consequences. Progress on a saturated benchmark is not progress, and a claimed improvement of a fraction of a point on one is not an improvement. And a benchmark's usefulness has a lifetime, so the ones worth attention are the ones currently discriminating, which changes.

Held-out and dynamic benchmarks address both problems at a cost. A held-out benchmark keeps the test set private, so it cannot be trained on — you submit a model and receive a score, and contamination is prevented structurally rather than hoped against. A dynamic benchmark changes its contents over time, adding new items, so it cannot saturate and old contamination stops mattering.

The cost, in both cases, is comparability: a private set cannot be inspected, so you cannot see whether its questions resemble your task; and a changing set means a score from last year and one from today are not the same measurement. Understanding that trade is what lets you read a result — a held-out score is more trustworthy and less interpretable, and a public score is the reverse.

Statistical significance is the third problem and it is the most ordinary. Two models differing by a small margin on a small test set may not differ at all. A benchmark of five hundred questions gives a margin of error of a few percentage points, so a two-point difference between models is inside the noise, and the ranking would change if the test set had been drawn differently.

Applying ordinary statistical reasoning to model comparison is rarer than it should be, and it is not difficult. Compute a confidence interval for each score from the number of items — a proportion's interval is elementary. If the intervals overlap substantially, report the two results as indistinguishable. Where the outputs are non-deterministic, run each model several times and account for that variation too, since the same model on the same set gives different scores.

And ask what a difference means before caring about it. A one-point difference in an accuracy that decides nothing is not worth a migration; a one-point difference in a rate of dangerous errors might be worth a great deal. The number and the consequence are separate questions and the report usually gives only the first.

Which brings the conclusion: build an internal benchmark you trust. It is the practical content of this topic, and everything above is the argument for it.

The properties that make it worth building. It is drawn from your tasks, with your inputs, your distribution and your definition of a correct answer — so its result predicts your outcome, which no public benchmark does. It is private, so it cannot be contaminated, which means never publishing it, never pasting it into a hosted model as an example, and being careful about where it is stored. It is versioned, so a number from six months ago can be attributed. It is small enough to read — a few hundred cases at most, since somebody must be able to look at the failures, which is where the information is. And it covers the cases you care about, including the rare and hard ones and the ones that should be refused, rather than being a random sample dominated by the straightforward majority.

Two habits keep it honest. Report a confidence interval alongside every score, so a small difference is visibly small. And refresh it periodically, adding recent real cases, because a fixed set that you have been choosing against for a year has quietly become the thing you optimised — which is the evaluation-overfitting warning from the language module, arriving here as the last word.

And the reading rule to take away. When a comparison is quoted at you, ask four questions: is the benchmark plausibly in the training data, is it saturated, is the difference larger than the noise, and does it measure anything resembling your task. Most quoted comparisons fail at least one, and the ones that survive all four are worth the attention.

What you should now be able to explain or do

Explain contamination, why it is usually accidental, and the signals suggesting it. Apply the training-cut-off rule when weighing a benchmark. Recognise saturation and say why progress on a saturated benchmark is not progress. State what held-out and dynamic benchmarks fix and what they cost in comparability. Compute and use confidence intervals, account for non-deterministic output, and separate the size of a difference from its consequence. Build an internal benchmark with the five properties. Report intervals and refresh the set. Apply the four reading questions.

Check yourself