1.10 Gradient descent and its variants

What this is and why it exists

Gradient descent is a two-line loop that trains everything from linear regression to language models — and a loop with famous failure modes. This topic gets you to implement it from scratch, meet the family (batch, stochastic, momentum, Adam), and read a loss curve the way a mechanic reads an engine note: diverging, crawling, or healthy.

The vocabulary

• Batch gradient descent — the true gradient over ALL data, once per step: exact, slow, memory-hungry.
• Stochastic (SGD) — the gradient of ONE example per step: cheap, noisy, surprisingly effective.
• Mini-batch — a few dozen to a few thousand examples per step: the practical default, noise tamed but not killed.
• Learning rate — the step-size multiplier on the gradient; the single most consequential number in training.
• Momentum — a running average of past gradients carries velocity through noise and down valleys; Nesterov looks ahead before correcting.
• AdaGrad / RMSProp / Adam — per-parameter step sizes adapted from each parameter's gradient history; Adam = RMSProp + momentum, the modern default.
• Divergence — loss climbing or exploding to NaN: steps too big for the local curvature.

The mental model

The loop is: measure the slope, step downhill, repeat — parameters minus learning-rate times gradient. Everything else in this topic is repair work on that loop's failures.

Noise, first. The full-batch gradient is exact but costs a pass over the whole dataset per step. A mini-batch estimates it from a sample — each step is wrong in a random direction but right on average, and you take a thousand of them in the time one exact step took. The noise even helps: it shakes descent off saddles and plateaus (ca3). This is the standard trade of the whole field: cheaper, noisier estimates, in volume.

Geometry, second. In the narrow valleys of ill-conditioned losses (ca1), raw descent zigzags — overshooting across the steep wall while inching along the valley floor. Momentum fixes it physically: keep a velocity, let gradients accelerate it — the across-the-valley components cancel over steps while the along-the-valley components add. Nesterov refines it with a look-ahead: evaluate the gradient where the velocity is about to take you. Adaptive methods attack the same problem per-coordinate: AdaGrad divides each parameter's step by the history of its gradients (steps shrink forever — its flaw); RMSProp makes that history a decaying average so steps stay alive; Adam adds momentum on top. Adam with its default settings is where sensible people start; SGD-with-momentum, tuned, still wins some final leaderboards.

Diagnosis, third — the loss curve is the instrument panel. Climbing or NaN: learning rate too high for the curvature; divide it by 10. Flat from the start: too low, or gradients not reaching the parameters (a bug — run the gradient check from ca1). Falling then plateauing: expected — consider a schedule that decays the rate. Wild oscillation: rate near the edge, or batch too small. Implementing the loop on a 2D bowl you can PLOT, then breaking it on purpose with a huge rate, teaches this panel faster than any table.

What you should now be able to explain or do

Implement the plain loop from scratch and watch it descend a plotted bowl. Choose batch, mini-batch or SGD for a stated dataset and defend it. Explain what momentum cancels and what Adam adapts. Read four loss-curve shapes to their causes.

Check yourself