7.7 Encoder models: BERT and family

What this is and why it exists

Encoder models read a whole sequence at once, in both directions, and produce a representation of it. For classification, extraction and ranking they remain the cost-effective choice, and fine-tuning one is cheap enough to do on a laptop-scale budget. The judgement this topic is really about is knowing when to reach for one instead of a general-purpose generative model — a decision that routinely changes a system's running cost by orders of magnitude at equal or better quality.

The vocabulary

• Encoder — a stack with no causal masking, so every position sees every other.
• Bidirectional — using context from both sides of a token.
• Masked language modelling — training by hiding tokens and predicting them from both sides.
• Sequence-level objective — an additional training task about the relationship between two segments.
• Head — a small task-specific layer added on top of the encoder.
• Pooling — reducing the per-token outputs to one vector for the sequence.
• Reranking — reordering a candidate list by scoring each candidate against the query.
• Distillation — training a smaller model to reproduce a larger one's behaviour.

The mental model

The training objective is what makes these models bidirectional. Hide a proportion of the tokens and ask the model to predict them from everything else — from the left and the right at once. That is only possible without causal masking, and it is exactly what a generative model cannot do, because a generative model must never see the future. The consequence is a representation of each token informed by its full context in both directions, which is precisely what you want for deciding what a text is about or which span is an entity, and not what you want for writing the next word.

The original models added a second, sequence-level objective about whether two segments followed one another. Later work found that this objective contributed little and that the recipe mattered much more, so it was dropped in favour of training longer on more data with better settings.

The successors improved along three different axes, and the distinction is worth holding because it tells you what kind of gain to expect from each. One line improved the training recipe — more data, longer training, larger batches, no second objective, better masking — and showed how much of the original result was recipe rather than architecture. A second line changed the attention mechanism, separating a token's content from its position so that the two contribute to attention scores separately, which improves handling of word order and relative distance. A third line shrank the model by distillation, training a small student to reproduce a large teacher's outputs, keeping most of the quality at a fraction of the size and latency. That third line is the one that matters most in production, and it is the least discussed.

Fine-tuning is cheap and the pattern is short. Take a pretrained encoder, add a small head, train the whole thing at a low learning rate for a few passes over your data. For classification the head reads a pooled representation of the sequence — either a designated summary position or an average over the tokens — and maps it to your classes. For extraction the head labels every token, which is the sequence-labelling machinery from earlier in this module. For span-based question answering the head predicts where the answer starts and where it ends. A few thousand labelled examples, a few minutes on a single accelerator, and you have a strong classifier, which is a very different proposition from anything requiring a large model.

The transfer-learning rules from the deep learning module apply directly: use the tokenizer that belongs to the checkpoint, keep the learning rate small, freeze nothing unless data is very scarce, and expect two to four passes to be enough — encoders overfit small datasets quickly, and more epochs generally makes things worse rather than better.

Now the judgement, which is the part worth being able to argue. For a fixed classification task with steady volume, compare the two options honestly. A fine-tuned encoder is small, runs in milliseconds, costs the same for every request, can be hosted on your own hardware so no text leaves your control, and — this is the part people miss — is frequently more accurate than a general-purpose model, because it was trained on your labels, your domain and your definition of the classes. A general-purpose generative model needs no labels and no training, adapts to a new class by editing a description, handles tasks nobody anticipated, and explains its answers. Its costs are latency, per-request expense that scales with volume, output that must be parsed and validated, sensitivity to how the request is phrased, and sending your text to wherever it runs.

So the working rule. High volume, stable label set, labels available or obtainable: fine-tune an encoder. Low or unpredictable volume, changing categories, no labels, or a task too open to specify: use the general-purpose model. And the strong middle option that combines them: use the general-purpose model to label a few thousand examples, then fine-tune an encoder on those labels and serve the encoder. You pay the large model once during development instead of on every request forever, and the resulting classifier is small enough to run anywhere. Check the labels by hand on a sample before trusting them, since you are inheriting the larger model's mistakes as ground truth.

One more place encoders earn their keep: reranking. Retrieval systems fetch candidates cheaply and imprecisely; a small encoder that reads the query and a candidate together and scores their match reorders the top few dozen far more accurately than the retrieval score alone. It is one of the highest-value, least-glamorous uses of a small model, and it appears again in the retrieval work later in the curriculum.

What you should now be able to explain or do

Say why the masked objective produces bidirectional representations and why a generative model cannot use it. Name the three axes along which the successors improved and what gain each represents. Fine-tune an encoder with a task head, choosing pooling appropriately and following the transfer rules. Argue both sides of the encoder-versus-generative choice on cost, accuracy, latency, data control and flexibility. State the working rule and the hybrid that combines both. Describe reranking and why a small model suits it.

Check yourself