7.7 Encoder models: BERT and family
You can fine-tune an encoder for classification cheaply.
Before:06. Deep LearningUnlocks:08. Large Language Models
Encoder models like BERT read whole sequences bidirectionally and remain the cost-effective choice for classification, extraction and reranking — fine-tuning one is cheap, fast and boringly reliable. It follows the transformer topic as its first specialisation. The default worth challenging is reaching for a generative LLM for everything: on high-volume classification a fine-tuned encoder is often better and an order of magnitude cheaper, and the invoice knows it.
Work through these
Masked language modelling and NSP
Training by hiding words and predicting them, plus an additional sentence-level objective in the original. These models read the whole text at once rather than left to right.
RoBERTa, DeBERTa, DistilBERT
Successors that improved the training recipe, changed the attention mechanism, or shrank the model while keeping most of its quality. Each represents a different kind of improvement.
Fine-tuning heads for classification and QA
Attaching a small task-specific head and training it, which is fast and needs relatively little data. This remains the cheapest route to a strong text classifier.
When an encoder beats an LLM on cost
For a fixed classification task, a small encoder is frequently faster, cheaper and more accurate than a general-purpose generative model. Knowing when to choose it is a genuinely valuable judgement.
LLM & NLP Course · Course
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.