1.10 Gradient descent and its variants
You can implement gradient descent from scratch and diagnose why it diverged.
Before:00. Orientation & SetupUnlocks:03. Data Handling & Analysis04. Classical AI — Agents, Search & Knowledge Representation
Gradient descent and its variants — momentum, RMSProp, Adam — are the engines that train every model in the rest of the course, and this topic has you implement them from scratch once so learning-rate trouble becomes visible instead of mystical. It sits at the end of the calculus arc as the payoff. The trap is reaching for Adam with defaults and no understanding; when it diverges, the from-scratch experience is what tells you whether the rate, the scaling or the data is at fault.
Work through these
Batch, stochastic and mini-batch descent
Using all the data, one example, or a small group per step trades noise against cost. The middle option is what nearly everything in practice uses, and understanding why is the item.
Learning rate, momentum, Nesterov acceleration
The step size, and the two ways of carrying information from previous steps to smooth the path. Momentum is what lets an optimiser cross a flat region instead of crawling.
Adaptive methods: AdaGrad, RMSProp, Adam
Methods that give each parameter its own effective step size, which is what makes training work without hand-tuning every dimension. They are the default in deep learning and they have known weaknesses.
Convergence intuition and divergence symptoms
What convergence looks like and what divergence looks like, so a failed run can be diagnosed rather than restarted with different numbers. The symptoms are distinctive once you have seen them.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.