6.20 Self-supervised and contrastive learning

What this is and why it exists

Labels are the bottleneck. You have far more data than you can afford to annotate, and self-supervised learning is how the field escaped that constraint: manufacture a task from the data itself, pretrain on everything, and fine-tune on the small labelled set you actually have. This is how modern pretraining works across every modality, and the practical payoff for you is direct — it is how to make use of the unlabelled data already sitting on your disk.

The vocabulary

• Pretext task — an invented task whose labels come from the data itself.
• Masked modelling — hiding part of the input and predicting it from the rest.
• View — one augmented version of an example.
• Positive pair — two views of the same example; negative — a view of a different one.
• Collapse — the failure where an encoder maps every input to the same representation.
• Momentum encoder — a slowly updated copy of the encoder, used to produce stable targets.
• Linear probe — a simple classifier trained on frozen representations, used to evaluate them.
• Zero-shot classification — classifying into categories the model was never trained on.

The mental model

A pretext task is a question the data can answer about itself. Hide part of an input and predict it; the answer is the part you hid. That is the whole idea, and its power is that the supervision is free and unlimited — every unlabelled example is a training example, as many times as you have ways to hide part of it.

Masked modelling is the form that dominates. In language, hide tokens and predict them from the surrounding context; the model must learn syntax, meaning and a good deal about the world to fill blanks well. In vision, hide patches and reconstruct them, which works better than it sounds because reconstructing a masked region requires understanding what the scene contains. In audio, mask spans of the signal. The key design decision is how much to hide. Too little and the answer is available from immediate neighbours, so the model learns local smoothing and nothing else. Too much and the task is impossible and the model learns the average. Vision needs a much higher masking fraction than language, because neighbouring pixels are far more redundant than neighbouring words — a fact worth remembering, because it explains why recipes do not transfer between modalities.

The contrastive family asks a different question: which two of these are the same thing? Take one image, make two augmented views of it, and require their representations to be close, while representations of views of different images are pushed apart. The augmentations are not incidental — they define what the model is being told to ignore. Crop and colour-jitter aggressively and the model learns that position and colour do not determine identity; leave the augmentations weak and it can satisfy the objective by comparing average colour and learn nothing.

Collapse is the characteristic failure and understanding it is the point of comparing the three influential approaches. If the objective only says "make views of the same image similar", mapping every input to the same constant vector satisfies it perfectly, and the encoder has learned nothing. Negatives are the classic defence: pushing different images apart makes the constant solution impossible, and the approach built on that needs a large number of negatives per step, which means very large batches. The second approach maintains a rolling store of recent representations produced by a slowly updated copy of the encoder, so many negatives are available without an enormous batch, and the slow copy keeps those targets stable rather than shifting under the model each step. The third approach dispensed with negatives entirely and still avoided collapse, using an asymmetry between the two paths — a predictor on one side and a slowly updated copy on the other — which was a genuinely surprising result and made clear that the essential ingredient was never the negatives themselves but some asymmetry that makes the constant solution unreachable.

Joint image and text training is the bridge between vision and language. Train an image encoder and a text encoder together so that an image and its caption land close in a shared space, while mismatched pairs are pushed apart, over a very large collection of image-caption pairs from the web. What emerges is a representation where language can address images. Classification then needs no training at all: write a description of each category, encode them, encode the image, and pick the nearest — categories the model was never explicitly trained on. That capability underpins a great deal of what is possible now, including how promptable segmentation and text-conditioned generation get their conditioning. It also inherits the character of its training data: web captions carry the biases and the gaps of the web, and the model's blind spots are its data's blind spots.

Then the evaluation protocol, without which none of this is measurable. A representation cannot be judged by its pretraining loss — a lower masked-prediction loss does not mean a more useful representation. The standard is the linear probe: freeze the encoder, train only a simple linear classifier on top of its outputs, and report that accuracy. Because the classifier can do almost nothing on its own, whatever it achieves is a property of the representation. Fine-tuning the whole encoder is the other protocol and it measures something different — how good a starting point this is, rather than how good the representation already is. Report which one you used, because the two numbers are not comparable, and a linear probe is what catches collapse: a collapsed encoder produces a probe accuracy at chance level, however healthy the pretraining loss looked.

The practical procedure. Pretrain on your own unlabelled domain data rather than reaching for weights from unrelated photographs — this is the answer to the domain-shift problem from the transfer topic. Choose masked modelling when your data is high-dimensional and redundant; choose a contrastive objective when you have a good notion of which transformations should not change identity. Check for collapse with a probe early and often. And then fine-tune on your small labelled set, where the whole exercise pays off.

What you should now be able to explain or do

Say what makes supervision free in a pretext task. Describe masked modelling and explain why the masking fraction differs by modality. Say what augmentations define in a contrastive objective. Explain collapse and how each of the three influential approaches avoids it, including what the third revealed. Describe joint image-text training and how zero-shot classification follows from it. Run and interpret a linear probe, and distinguish it from fine-tuning as an evaluation. Choose an objective for your own unlabelled data and check for collapse.

Check yourself