7.1 Text preprocessing and tokenization
You can turn raw text into model input and back.
Before:06. Deep LearningUnlocks:08. Large Language Models
Tokenization is the boundary between text and model: normalisation choices upstream change everything downstream, and subword schemes — BPE, WordPiece, SentencePiece — are how modern models handle open vocabularies. Token counts are also what API bills are denominated in. It opens the NLP module because nothing in it runs on raw strings. The silent killer is mismatched preprocessing: tokenize at inference differently than at training and quality degrades with no error message anywhere.
Work through these
Normalization, casing, stopwords, stemming, lemmatization
Standardising case, removing common words, and reducing words to a root form by rule or by dictionary. These classical steps matter less with modern models and still matter for classical ones.
Word, character and subword tokenization
Splitting text into units, at word level, character level, or something in between. The in-between option is what all current large models use.
BPE, WordPiece, SentencePiece
Three algorithms that learn a vocabulary of subword pieces from data. Knowing they exist explains why models split unfamiliar words into odd-looking fragments.
Vocabulary size, OOV and the token-cost link
How large the vocabulary is, what happens to words outside it, and the direct link between how text is split and what a request costs. That last connection matters commercially as well as technically.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.