7.11 Evaluating NLP systems
You can tell whether a text system got better or just different.
Before:06. Deep LearningUnlocks:08. Large Language Models
Evaluation separates better from different: automatic metrics are cheap and weakly correlated with quality, human evaluation is the gold standard and needs protocols, and golden test sets are what make regressions visible. It closes the NLP module because every earlier topic produced things needing judgement. The quiet failure is eval-set overfitting — iterate against the same small set long enough and improvement on it stops meaning improvement anywhere else.
Work through these
Automatic metrics and their weaknesses
Automatic measures are convenient, comparable and frequently disagree with human judgement. Knowing what each measure actually rewards is what keeps a report honest.
Human evaluation protocols
Structured human judgement, with the protocol decided before any results are seen. Without a protocol, evaluation becomes an argument about impressions.
Annotation guidelines and inter-annotator agreement
Written labelling instructions and the measurement of how much annotators agree. Low agreement means the task is underspecified, not that the annotators are careless.
Building a golden test set
A carefully constructed test set that changes rarely is what makes improvements comparable over time. Building it early is far easier than reconstructing it later.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.