7.5 Text classification end to end

What this is and why it exists

This is the whole loop, shipped: a text classifier with an evaluation you would defend to somebody who wanted it to be wrong. The technical parts you already have. What this topic adds is the practice that produces most of the improvement in real projects — reading the actual mistakes, grouped by kind — and the discovery that changes how you interpret every number afterwards: some of what looks like model error is disagreement about what the right answer was.

The vocabulary

• Sentiment — how positive or negative a text is.
• Topic — what a text is about, from a fixed set.
• Intent — what the writer wants to happen.
• Distribution shift — the data changing after the model was trained.
• Error analysis — reading and grouping the mistakes.
• Annotation noise — inconsistency in the labels themselves.
• Inter-annotator agreement — how often independent labellers give the same label.
• Ceiling — the best performance achievable given the label quality.

The mental model

The three common tasks differ in how stable their labels are over time, which decides how much maintenance each will need. Sentiment is comparatively stable: what counts as an angry message does not change much. Topic labels are stable until the world adds a topic, which it does. Intent is the least stable of the three — it depends on how people phrase requests, and phrasing changes as your product changes, as a competitor introduces a term, or as a new problem appears in the world. An intent classifier is a thing you maintain, not a thing you ship, and planning for periodic relabelling from the start is the difference between a system that degrades gracefully and one that quietly stops working.

Imbalance behaves as it did in the classical module, with one added constraint. Fix the metric, tune the threshold, weight the classes, and only then resample — all of that carries over. What is different is that text examples for a rare class are expensive to obtain, because somebody has to read and label them. So two responses become disproportionately valuable. Targeted collection: search the unlabelled pool for text resembling the rare class and label those, rather than labelling at random and finding one positive in two hundred. And grouping: several rare classes that need the same downstream action can be one class, which frequently turns three hopeless problems into one workable one. Ask what the prediction is for before deciding how many classes there are — a distinction nobody acts on differently is a distinction costing you data.

Now error analysis, which is where the real learning happens. Take every mistake on the validation set, read them, and sort them into groups by what went wrong. Not by class — by cause. Typical groups: the label is wrong; the text is genuinely ambiguous; the model missed a negation; the text is in a language or register the training data lacks; the example is a template or boilerplate that dominates by volume; the true class is present but so is another. Count each group.

That count is a plan. If a third of the errors are wrong labels, your next move is annotation, not modelling. If a quarter are negation, add pair features or a model that reads order. If a fifth are one narrow template, a rule handles it precisely and removes it from the model's job. The aggregate score tells you how well you are doing and nothing about what to do next; fifty read examples tell you exactly what to do next. An hour spent this way routinely beats a week of hyperparameter search, and it is the habit that most separates people who ship working classifiers from people who tune.

Which leads to the assumption this topic exists to break: labels are not truth. They are one or more people's judgement, recorded under time pressure, from guidelines that were probably ambiguous. Have two people independently label two hundred of the same examples and measure how often they agree. Their agreement is approximately the ceiling on what any model can be scored at — if two careful humans agree 85 percent of the time, a model reported at 92 percent against one of them is not better than human, it has learned that particular annotator's habits.

Low agreement means the task is underspecified, not that the annotators were careless. That is the professional reading, and it points at the fix: go and look at the cases they disagreed on, decide what the right answer is, and write it into the guidelines with examples. Do that and both the humans and the model improve, because you have removed the ambiguity rather than trained around it. Doing this before scaling annotation is much cheaper than doing it after, when the disagreement is already baked into thousands of labels.

So the practical procedure that separates the two kinds of error. Take a sample of the model's mistakes and relabel them blind — without seeing what the model said or what the original label was. Some will come back matching the model, which means the original label was wrong and the model was right. That proportion tells you what share of your error rate is annotation noise, and it is routinely a quarter or more. Report it. A model at 88 percent against labels that are themselves 90 percent consistent is a different result from a model at 88 percent against near-perfect labels, and saying which one you have is what makes the evaluation honest.

The shipping checklist, then. A frozen test set, split by time and not at random, so the score reflects use rather than memory. The simple baseline reported alongside the model. Entity-level or class-level breakdowns, not only the overall number. The confusion matrix, so the pattern of mistakes is visible. A statement of the annotation agreement and the estimated share of error that is label noise. And a plan for the periodic relabelling the task will need. That is a shipped classifier with an honest evaluation, and each item on the list exists because leaving it out has misled somebody.

What you should now be able to explain or do

Say which of the three tasks ages fastest and what follows for maintenance. Apply the imbalance responses under the constraint that text labels are expensive, using targeted collection and class grouping. Run an error analysis grouped by cause and turn the counts into a plan. Measure inter-annotator agreement and interpret it as a ceiling. Read low agreement as an underspecified task and fix the guidelines. Estimate the share of error that is annotation noise by blind relabelling. Ship with the full evaluation checklist.

Check yourself