7.4 Sequence labelling

What this is and why it exists

Sequence labelling puts a label on every token, and it is the machinery behind turning unstructured documents into rows in a table — names, dates, amounts, product references, pulled out of prose. The models are approachable. The part that trips people is the evaluation: score at token level and a system that half-matches every entity looks respectable while being useless, because a half-matched name is not a name.

The vocabulary

• Token — one unit of text carrying one label.
• Part-of-speech tag — the grammatical role of a word.
• Chunking — grouping adjacent words into phrases.
• Named entity — a span of text naming a person, place, organisation, date or similar.
• BIO tagging — a scheme marking whether a token begins an entity, is inside one, or is outside any.
• Span — a contiguous run of tokens forming one entity.
• Structured prediction — predicting a whole sequence of labels at once, not each independently.
• Transition score — how plausible one label is immediately after another.

The mental model

The classical labelling tasks come first because everything else reuses their shape. Part-of-speech tagging assigns each word its grammatical role, which disambiguates words that are both noun and verb and gives downstream rules something to work with. Chunking groups adjacent words into phrases — noun phrases especially — which is a cheap and surprisingly effective way to find candidate entities before any model is involved. Neither is glamorous, and both remain useful as features and as a way to sanity-check that a pipeline is reading the text as you expect.

Named entity recognition is the task most people arrive for, and its labelling scheme is the idea to take away. An entity is a span, and per-token labels have to encode span boundaries somehow. The BIO scheme does it with three prefixes: a token that begins an entity, a token inside one, and a token outside any. Two entities of the same type sitting adjacent stay distinct because the second one starts with a begin tag. That trick — encoding spans as per-token labels — appears far beyond this task, anywhere a model must mark regions of a sequence.

The scheme also creates label sequences that are impossible: an inside tag for one entity type cannot follow a begin tag of a different type, and an inside tag cannot follow an outside tag at all. A model that labels each token independently will produce these, because nothing tells it not to.

Which is exactly what structured prediction is for. A conditional random field scores a whole label sequence rather than each label alone, combining how well each label fits its token with how plausible each label is immediately after the previous one. Impossible transitions get a score so low they are never chosen, and the best complete sequence is found by dynamic programming rather than by taking the best label at each position. The result is consistent output, and this remains a standard finishing layer on top of a neural tagger for precisely that reason — the network judges tokens, the transition layer enforces that the sequence makes sense.

Then the evaluation point, which is the whole reason this topic exists. Token-level scoring counts each token's label independently. Entity-level scoring counts an entity as correct only if its complete span and its type both match. These diverge dramatically. Consider a three-word organisation name where the model tags the first two words and misses the third: token-level scoring records two hits out of three and looks good; entity-level scoring records one miss and one false entity, and it is right to, because a truncated organisation name is wrong in every use you would put it to.

Report entity-level precision, recall and their harmonic mean, and report them per entity type. The per-type breakdown is where the actionable information is — a system that handles people and places well and dates badly needs work on dates, and the overall number hides that completely. When a published result seems implausibly strong, checking whether it was scored at token or entity level explains it more often than not.

Two further evaluation details worth knowing. Decide in advance whether a correct span with the wrong type counts as partly right, and say which convention you used, because both exist. And boundary conventions — whether a title, a suffix or a trailing possessive belongs inside the span — must be written into the annotation guidelines, or your annotators will disagree and the model will learn the disagreement.

A production library makes the practical side short. The spaCy documentation describes the arrangement: "when you call nlp on a text, spaCy will tokenize it and then call each component on the Doc, in order", and "the pipeline used by the trained pipelines typically include a tagger, a lemmatizer, a parser and an entity recognizer". The tokenizer is deliberately outside that sequence — "the tokenizer is a 'special' component and isn't part of the regular pipeline", because "there can only really be one tokenizer, and while all other pipeline components take a Doc and return it, the tokenizer takes a string of text and turns it into a Doc". Components can be disconnected when you do not need them, which matters: running a parser you never read is a large share of the processing time on a big corpus.

The practical route from text to structured output, then. Start with the pretrained pipeline and measure it at entity level on your own data. Add rule-based matching for the entities that are patterns rather than language — reference codes, dates in a fixed format, amounts — because a rule is exact, fast and explicable, and a model is none of those for a task a rule solves. Then train or fine-tune only for the entity types your domain has and the pretrained model does not. Most extraction projects need much less model than they first assume, and much better annotation guidelines than they first write.

What you should now be able to explain or do

Say what tagging and chunking give you and why they remain useful. Explain the BIO scheme and how it keeps adjacent entities distinct. Say why independent per-token prediction produces impossible sequences and what a transition-scoring layer changes. Distinguish token-level from entity-level scoring and give the example that separates them. Report per-type entity-level scores and say what the breakdown reveals. Describe how a production pipeline is ordered and why the tokenizer sits outside it. Choose rules over a model where the entity is a pattern.

Check yourself