7.11 Evaluating NLP systems

What this is and why it exists

"Better" and "different" look identical from inside a project, and telling them apart is the whole job of evaluation. Automatic measures are cheap and correlate weakly with what people actually want. Human judgement is the standard and needs a protocol or it becomes an argument about impressions. And the set you keep testing against slowly stops measuring anything, because you have been optimising it. This topic is how to build an evaluation you can still trust in six months.

The vocabulary

• Automatic metric — a score computed without human involvement.
• Reference-based — scored by comparison with a human-written correct answer.
• Reference-free — scored without one, usually by another model.
• Protocol — the rules of a human evaluation, fixed before results are seen.
• Annotation guidelines — the written instructions labellers follow.
• Inter-annotator agreement — how often independent labellers give the same answer.
• Golden test set — a carefully built set that changes rarely.
• Regression — a change that made something previously working worse.

The mental model

Automatic metrics are development instruments, not evidence of quality, and the skill is knowing what each one actually rewards. Overlap-based measures reward sharing words with a reference and are blind to fluency and to meaning-reversing errors. Embedding-based measures reward semantic similarity and are more forgiving of rephrasing, at the cost of being less interpretable and inheriting whichever model produced the embeddings. Model-as-judge scoring — asking a strong model to rate an output — correlates better with human judgement than the older measures and carries its own biases: a preference for longer answers, for a familiar style, and for outputs resembling what the judge would have written. State what each measure rewards when you report it, because a number without that is an invitation to be misled by it.

Two rules keep automatic numbers honest. Compare only within one test set with one set of references and one preprocessing, since these numbers are not comparable across those choices. And check the direction of any large improvement by reading a sample of the outputs — a jump usually means something changed about the format or the length rather than about the quality.

Human evaluation is the standard, and it needs a protocol decided in advance. Write down what you are judging, on what scale, with what instructions, on which examples, before any results exist. Otherwise the evaluation becomes a discussion of impressions in which whoever built the system remembers the good examples.

Four design choices carry most of the weight. Comparison beats rating: people are much more consistent asking "which of these two is better" than assigning a number out of five, so prefer side-by-side comparison with the order randomised and the source of each output hidden. Judge one thing at a time: fluency, factual accuracy and helpfulness are separate questions and combining them into one score loses all three. Sample deliberately: random examples are dominated by the common case, so include a stratified sample covering the situations you care about, and report them separately. And measure the judges: give several people the same examples and compute their agreement, because an evaluation whose annotators disagree is not measuring the system.

Low agreement means the task is underspecified, not that the annotators were careless. This is the professional reading and it points directly at the fix. Take the disagreements, work out what the right answer should be, and write that into the guidelines with worked examples — especially the edge cases, which is where guidelines are always thin. Good guidelines are built from disagreements rather than written in advance and hoped over. Do this before scaling annotation, because after ten thousand labels the ambiguity is baked in and re-labelling is far more expensive than agreeing.

The golden test set is what makes improvement comparable over time, and building it early is much easier than reconstructing it later. It should be carefully checked rather than large, cover the situations that matter including the rare and hard ones, be split by time from the training data rather than at random, and be versioned so you can say which version a past number came from. Keep it out of every training pipeline, and if the system is built on published models, be alert to the possibility that a public benchmark is already in their training data — which is a strong argument for building your own set from your own material.

Then the failure this topic exists to prevent. Every decision you make against a test set fits it a little more closely. Run fifty experiments against the same two hundred examples and the winner is partly better and partly lucky, and the gap between your reported number and real performance grows silently the whole time. The evidence you have crossed the line is familiar: improvements that do not survive contact with users, differences smaller than the variation between annotators, and a leaderboard where reordering the top entries takes very little.

Three defences. Keep a held-out set that is used rarely — quarterly, or at a release — and treat that number as the honest one, while a smaller development set carries daily iteration. Refresh the development set periodically, adding new examples from recent data, so it does not become a fixed target you have memorised. And compare improvements against the noise: compute the variation across annotators or across resampling of the test set, and treat any gain smaller than that as a tie, preferring the simpler or cheaper system.

Two habits complete it. Keep a regression suite of specific cases that must not break — the bug somebody reported, the input that used to crash, the phrasing that produced an embarrassing answer — and run it on every change; that is what catches the improvement that quietly broke something else. And record, alongside every reported number, the test set version, the metric settings, and the exact model and preprocessing used. A number without those cannot be compared with anything, including your own result from last month.

What you should now be able to explain or do

Say what each family of automatic metric rewards and what it is blind to, including the judge model's biases. Keep automatic comparisons within one set, one reference and one preprocessing. Design a human protocol before seeing results, using comparison, single-dimension judgements, stratified samples and measured agreement. Read low agreement as underspecification and rebuild the guidelines from the disagreements. Build a golden test set with the right properties and keep it out of training. Recognise having overfitted an evaluation set and apply the three defences. Maintain a regression suite and record what every number was measured with.

Check yourself