7.1 Text preprocessing and tokenization

What this is and why it exists

Tokenization is the boundary between text and model, and everything on the model side inherits whatever you decided on the text side. Get it wrong at inference in a way that differs from training and the model degrades quietly, with no error and no warning — one of the most common silent failures in applied language work. This topic also explains why models split unfamiliar words into odd-looking fragments, and why the way text is split is the same thing as what a request costs.

The vocabulary

• Normalisation — standardising text before splitting: case, accents, whitespace, punctuation.
• Stemming — chopping words to a root by rule; lemmatisation — reducing to a dictionary form.
• Stopwords — very common words sometimes removed as uninformative.
• Token — one unit of model input.
• Subword — a unit between a character and a word.
• Vocabulary — the fixed set of tokens a model knows.
• Out-of-vocabulary — text the vocabulary cannot represent directly.
• Round-trip — encoding to tokens and decoding back to text.

The mental model

Every choice upstream changes everything downstream, so make them deliberately. Lowercasing collapses a distinction the model can then never recover — helpful for topic classification, harmful where names and acronyms matter. Stripping punctuation removes sentence boundaries and, with them, negation cues and question marks. Removing stopwords was standard practice for count-based methods, where those words add dimensions and no information, and it is actively harmful for modern models, where "not" and "but" carry much of the meaning. The old normalisation steps matter for the classical methods in the next topic and mostly do not for anything built on subword models, which is the single most useful thing to know about them.

Stemming and lemmatisation both reduce word forms to one, differing in method and cost. Stemming applies rules to chop endings, which is fast and produces fragments that are not words. Lemmatisation uses a dictionary and grammatical role to produce the proper base form, which is slower and correct. Both exist to stop a count-based model treating a word and its plural as unrelated features; neither is needed when the representation already handles morphology.

The three levels of splitting are a trade with an obvious middle. Word-level splitting gives units that mean something, and the vocabulary becomes enormous because every variation needs its own entry — the tokenizer documentation notes that this makes the embedding matrix enormous and that words outside the vocabulary map to an unknown token, "so the model can't handle new words". Character-level splitting has the opposite profile: "the vocabulary is small and every word can be represented, so there's no unknown problem. But sequences become much longer" and each unit "carries far less meaning", so performance suffers.

Subword splitting takes the middle. The documentation describes it as splitting "text into units between words and characters, keeping the vocabulary compact while still capturing meaningful pieces", where "common words stay intact as single tokens, and rare or unknown words decompose into subwords". That is why an unusual name arrives as three fragments while an ordinary word arrives whole — not a defect, but the mechanism that lets a fixed vocabulary represent text it has never seen.

Three algorithms learn that vocabulary from data, and their differences are instructive. Byte-pair encoding starts from individual characters and "iteratively merges the most frequent adjacent pair", continuing "until it reaches the target vocabulary size, which equals the base vocabulary size plus the number of merges". Its byte-level variant uses the 256 byte values as its base "instead, ensuring every word can be tokenized without the unknown token" — which is how any character in any script becomes representable without an unmanageable base vocabulary.

WordPiece merges from the bottom up as well but chooses differently: it "merges pairs that maximize the likelihood of the training data", scoring a pair by its joint frequency against the product of its parts. The documentation draws the contrast directly — the other approach "merges whichever pair appears the most", while this one "measures how informative each merge is. Two tokens that appear together far more than chance predicts get merged first."

SentencePiece solves a different problem: it "is a tokenization library that applies BPE or Unigram directly on raw text", because the others "assume whitespace separates words, which doesn't work for languages like Chinese and Japanese that don't use spaces". It treats input as a raw stream and puts the space character into the vocabulary as a visible marker, restoring it on decoding — which is also what makes encoding and decoding exactly reversible.

Vocabulary size is a real trade. Larger means fewer tokens per document, shorter sequences, and cheaper attention, at the cost of a bigger embedding table and rarer tokens that are poorly learned. Smaller means the reverse. And the connection to cost is direct: a request is priced and limited by token count, not by character count, so how text splits decides what it costs. This falls unevenly — text in scripts under-represented in the vocabulary's training data splits into many more tokens per unit of meaning than English does, so the same content costs more and consumes more of the available context. It is worth measuring on your own data rather than assuming.

Finally, the rule that prevents the silent failure. The tokenizer is part of the model, not a preprocessing choice you make separately. Load the one that came with the checkpoint, apply the same normalisation you applied in training, and test the round trip: encode a sample, decode it, and compare with the original. A mismatch in casing, in whitespace handling, or in which tokenizer was loaded degrades results by a few points and reports nothing at all.

What you should now be able to explain or do

Say what each classical normalisation step costs and where it still applies. Distinguish stemming from lemmatisation. Give the trade between word, character and subword splitting in the documentation's own terms. Describe how the merge-based and likelihood-based algorithms differ in what they merge. Say what the raw-stream library solves and why decoding is exact. Explain the vocabulary-size trade and the direct link to cost, including who it falls hardest on. Pair a tokenizer with its checkpoint and verify with a round trip.

Check yourself