8.12 Evaluating LLM systems

What this is and why it exists

Shipping on the strength of a few impressive outputs is the default in this area, and it does not survive. Evaluation is what lets you say a change made the product better rather than different, and the machinery is not complicated — a fixed set of cases, a suite that runs on every change, and where a model does the grading, evidence that its grades agree with a person's. This topic is that machinery, and the specific ways an unvalidated judge can invert your conclusions.

The vocabulary

• Golden set — a fixed set of cases with agreed-good answers.
• Regression suite — cases that must not break, run on every change.
• Model-as-judge — using a model to grade another model's output.
• Position bias — a judge preferring whichever answer it sees first.
• Verbosity bias — a judge preferring longer answers.
• Task metric — how well the system does your job.
• Offline evaluation — measured before release; online — measured in production.
• Shadow traffic — running a new version alongside the old without showing it.

The mental model

The golden set is the foundation and everything else is an addition to it. Fifty to two hundred cases, drawn from real use rather than invented, each with an agreed-good answer and the reasoning for why it is good. Cover the common path, the hard cases, the edge cases and the ones that should be refused. Keep it under version control, keep it out of every prompt and every training set, and version it so a number from three months ago can be attributed to a set.

Two properties make it worth the effort. It is fixed, so two numbers are comparable — the whole point. And it is small enough that a person can read the failures, which is where the actionable information is, exactly as the text-classification topic argued.

The regression suite is the golden set's sharper half: cases that must not break, whatever else changes. The bug somebody reported. The input that produced an embarrassing answer. The phrasing that leaked the system prompt. The document that made the parser fail. Each becomes a permanent case. Since these systems have no type system, this suite is the only thing that catches a prompt clarification that fixed one situation and broke another, and it should run automatically on every change to a prompt, a model, a retrieval setting or a schema — all four are code, and all four cause regressions.

Assert on properties rather than exact strings, for the reason the decoding topic gave: identical inputs do not reliably produce identical outputs. Does it parse. Does it satisfy the schema. Is the required fact present. Is the forbidden thing absent. Is a refusal present where one is required. Are the citations real. Those assertions are stable across the variation you cannot remove.

Model-as-judge is what makes evaluation affordable at any scale, and it must be validated before it is trusted. The pitch is straightforward: a strong model grades outputs against a rubric, thousands of times, for a small cost, and it correlates with human judgement better than the older automatic measures. The catch is that it is not neutral.

The biases are documented and specific. Position bias: given two answers to compare, judges favour one position over the other — so present each pair in both orders and average, and if the verdict flips with the order you have measured the judge, not the answers. Verbosity bias: longer, more thorough-looking answers score higher regardless of correctness, so a change that made the model more verbose will look like an improvement. Self-preference: a judge scores outputs resembling its own style more kindly, which matters when the judge and the system share a family. Rubric drift: without a specific rubric, a judge grades on general impressiveness, which is not what you asked.

So validate before trusting, and the validation is the whole ballgame. Have people grade a hundred cases. Have the judge grade the same hundred. Measure the agreement, and compare it against how much two people agree with each other — because that human agreement is the ceiling, exactly as in the text-classification topic. A judge agreeing with people about as often as they agree with each other is usable. One that does not is measuring something else, and every number it produces afterwards is that something else. Re-validate whenever you change the judge, its prompt or the rubric, and record the agreement figure beside any result the judge produced.

Task metrics versus benchmark scores is the substitution to refuse. Public benchmark performance tells you something general about a model; it does not tell you how it does your task on your data with your prompt. The two come apart routinely, and a model that scores better on a public leaderboard can be worse for you. Report what your system does on your golden set. Where a benchmark number is relevant, report it as context, labelled, and never in place of the thing you were actually asked.

Three kinds of evaluation, answering three different questions, and a serious system uses all three.

Offline runs the golden set and the regression suite before release: fast, repeatable, cheap, and it answers "did I break anything and did the thing I changed improve". It cannot tell you what real users will do.

Online measures the deployed system: task completion, retries, escalation to a person, thumbs, abandonment, latency, and cost. It answers "is this actually working for people", which offline cannot, and it is slow and noisy and confounded by everything else happening.

Shadow traffic runs the new version on real requests alongside the old without showing anyone its output. Real inputs, real distribution, no user risk — it answers "how would this behave on real traffic" before anyone sees it, and it is the cheapest way to discover that a change fine on your golden set falls over on the inputs people actually send.

The habit that ties it together. Before making a change, write down what you expect it to improve and by how much. Then measure. A change that improved nothing measurable is a change to revert, however sensible it seemed — and having written the prediction down beforehand is what stops the result being reinterpreted as a success afterwards.

What you should now be able to explain or do

Build a golden set with the right size, sources and properties, and keep it out of prompts and training. Maintain a regression suite from real incidents and run it on all four kinds of change. Assert on properties rather than exact strings, and say why. Set up a judge with a specific rubric, name its four biases, and control for position by presenting both orders. Validate a judge against human agreement, compare with the human-to-human ceiling, and re-validate on change. Refuse the benchmark-for-task-metric substitution. Say what offline, online and shadow evaluation each answer. Predict before measuring.

Check yourself