8.12 Evaluating LLM systems
You can prove a change made the product better, not just different.
Before:07. Natural Language ProcessingUnlocks:09. Agentic AI12. Frontier Topics
LLM evaluation is how you prove a change made the product better rather than merely different: golden sets, regression suites, and judge models validated against human judgement before being trusted. It sits near the module's end because every statement of improvement routes through it. Shipping on vibes is the named enemy — a few impressive outputs generalise poorly, and an unvalidated judge model carries position and verbosity biases that quietly reward the wrong things.
Work through these
Golden sets and regression suites
A fixed set of cases with known good answers, run on every change. This is the foundation and everything else is an addition to it.
LLM-as-judge: setup, bias, and validation against humans
Using a model to grade output, which is fast and carries its own biases toward length and style. It must be validated against human judgement before it is trusted.
LLM & NLP Course · CourseTask metrics vs. benchmark scores
How well a system does your task is not the same as how it scores on public benchmarks. Reporting the second in place of the first is a common and misleading substitution.
Offline eval, online eval, and shadow traffic
Testing before release, measuring in production, and running a new version alongside the old without showing it to users. The three answer different questions and a serious system uses all three.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.