6.5 Optimizers and learning-rate schedules
You can get a stuck model training again.
Before:05. Classical Machine LearningUnlocks:07. Natural Language Processing11. The ECE Bridge — Signals, Edge & Embedded AI
Optimisers and schedules separate a model that trains from one that stalls: AdamW as the modern default, warmup to survive the early chaos, cosine decay to land well, clipping to tame spikes. It sits mid-craft because it consumes the gradients earlier topics produced. The portability myth is the trap — a learning rate copied between models or batch sizes is the single most sensitive number in the whole configuration, and it does not travel.
Work through these
SGD, momentum, Nesterov
The base optimiser and the two ways of carrying momentum from previous steps. Everything more sophisticated is a variation on this.
Adam, AdamW and weight-decay decoupling
The adaptive optimiser that is the practical default, and the corrected variant that applies weight decay properly. The difference between them is small to state and matters in practice.
Warmup, cosine decay, one-cycle
Changing the step size over the course of training, starting small, then following a curve down. Schedules frequently matter more than the choice of optimiser.
Gradient clipping
Capping the size of a gradient step prevents a single bad batch destroying a run. It is a one-line safeguard that is standard in sequence models.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.