6.14 Transfer learning and fine-tuning

What this is and why it exists

Transfer learning is the working default, not an advanced technique. Starting from weights somebody else trained on a very large dataset, strong accuracy on five hundred labelled images is routine; from random weights on the same five hundred it is out of reach. This topic is how to do it in the right order, because doing it in the wrong order destroys the very thing you started from — and how to recognise the case where transfer will not help at all.

The vocabulary

• Pretrained weights — parameters learned on a large dataset, published for reuse.
• Backbone — the pretrained part that produces features.
• Head — the small new part that maps features to your classes, trained from scratch.
• Feature extraction — freezing the backbone entirely and training only the head.
• Fine-tuning — continuing to train some or all of the backbone.
• Freezing — holding parameters fixed so no gradient updates them.
• Discriminative learning rates — different step sizes for different layers.
• Domain shift — a difference between the data the weights came from and yours.

The mental model

Why it works is the hierarchy from the first topic. Early layers learn edges, colours and textures, which are not properties of the dataset they were trained on but properties of images. Middle layers learn parts and patterns. Only the last layers are specific to the original categories. So the early and middle layers are reusable almost without qualification, and the amount you should retrain increases as you move deeper. How much you fine-tune is a function of how much data you have and how far your images are from the originals, and that single sentence answers most questions in this topic.

The two ends of the range. Feature extraction freezes the backbone completely and trains a new head on its outputs; it is fast, it needs very little data, it cannot overfit much because almost nothing is being learned, and it is the right choice with a few hundred images or with images that resemble the pretraining data. Full fine-tuning continues training everything at a small learning rate; it reaches higher accuracy, needs more data — a few thousand at least — and will overfit given less. In between sits the common answer: freeze the early layers, fine-tune the later ones.

The order matters, and getting it wrong is the trap. A freshly initialised head produces meaningless outputs, so the first gradients are large and arbitrary. Let those flow into the backbone at a normal learning rate and they scramble the pretrained features before the head has learned anything worth backpropagating — the model recovers slowly, if at all, and ends below what feature extraction alone would have given. The correct sequence is: freeze the backbone, train the head until it is doing something sensible, then unfreeze progressively, at a much smaller learning rate. Unfreeze from the top down, since the last layers are the ones needing the most change.

Discriminative learning rates make the second phase work. Rather than one rate for everything, give the head the largest, the last backbone block something several times smaller, and each earlier group smaller still. The early layers barely move — which is what you want, since they were already right — while the later ones adapt. This is what makes fine-tuning on a few hundred examples viable rather than a way of forgetting a large dataset.

Three settings ride along with this. Keep the learning rate for fine-tuning well below what you would use from scratch, because you are adjusting rather than searching. Watch the normalisation layers: if the batch is small, the running estimates from pretraining can be better than what your data provides, and freezing them is a reasonable choice. And use the same input preprocessing the weights were trained with — the same resizing convention and the same normalisation constants — because a mismatch here silently degrades everything and is very frequently the cause of disappointing transfer.

Domain shift is where transfer fails, and knowing the failure prevents a confident waste of time. Weights trained on everyday photographs transfer well to other everyday photographs, reasonably to product shots and satellite imagery, and poorly to medical scans, microscopy, radar or spectrograms, where the statistics of the images are unlike anything in the original data. When the source and target are far apart, the early layers are still worth something — edges are still edges — but the middle and later ones are not, so fine-tune more of the network and expect to need more data. If the domain is truly distant, pretraining on unlabelled data from your own domain, using the self-supervised methods later in this module, transfers better than pretraining on unrelated photographs. And transfer can actively hurt: a strongly pretrained model can carry biases from its source data into a domain where they are wrong, so measure against a from-scratch baseline rather than assuming.

The model zoos are the normal starting point. Large public collections and libraries make thousands of pretrained backbones available behind a consistent interface, so swapping architectures is a change of one string. Two habits when using them. Record exactly which weights you used and where they came from, because the same architecture name covers many different sets of weights trained on different data with different recipes, and a result that cannot name its starting point cannot be reproduced. And read the licence and the description of the training data — pretrained weights carry both legal terms and the characteristics of the data they came from, and both travel into your model.

What you should now be able to explain or do

Explain why early layers transfer and later ones do not. Choose between feature extraction and full fine-tuning from data volume and domain distance. Carry out the freeze-then-unfreeze sequence in the right order and say what goes wrong if reversed. Set discriminative learning rates sensibly. Match the preprocessing to the pretrained weights and say why a mismatch is costly. Recognise domain shift, say where transfer helps little, and name the alternative. Use a model zoo, recording the exact weights and checking the licence and source data.

Check yourself