6.20 Self-supervised and contrastive learning
You can pretrain on unlabelled data you already have.
Before:05. Classical Machine LearningUnlocks:07. Natural Language Processing11. The ECE Bridge — Signals, Edge & Embedded AI
Self-supervised learning manufactures labels from data itself — masked tokens, contrastive pairs — and is how modern pretraining escaped the labelling bottleneck; CLIP's joint image-text space is what makes zero-shot classification possible at all. It sits late in the module because it presumes the architectures. The failure mode with a name is collapse: without the right asymmetries a contrastive setup maps everything to one point, achieving perfect agreement about nothing.
Work through these
Pretext tasks and masked modelling
Inventing a task from the data itself, such as predicting hidden parts, so no labels are needed. This is what unlocked training on data nobody could afford to annotate.
SimCLR, MoCo, BYOL
Three influential approaches to learning by comparing views of the same and different examples. Their differences are instructive about what the objective actually needs.
CLIP and joint image–text embedding
Training images and their captions into a shared representation, which enables classification into categories never explicitly trained on. It is the bridge between vision and language.
Evaluating representations: linear probes
Judging a learned representation by training a simple classifier on top of it, which isolates the representation from the classifier. It is the standard protocol and worth knowing before reading results.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.