6.3 Activations and initialization
You can explain why a net stopped learning at epoch three.
Before:05. Classical Machine LearningUnlocks:07. Natural Language Processing11. The ECE Bridge — Signals, Edge & Embedded AI
Activations and initialisation decide whether gradients survive the trip through depth — the ReLU family against saturating curves, He against Xavier scaling. When a network flatlines at epoch three, the cause usually lives in this topic. It sits early in the training-craft run because these choices precede training. The symmetry lesson is absolute: initialise weights to zeros or equal values and every unit computes the same thing forever, because nothing ever breaks the tie.
Work through these
ReLU, LeakyReLU, GELU, SiLU, tanh
The nonlinear functions in common use, differing in how they behave for negative inputs and how smoothly they bend. The choice affects training speed more than final accuracy.
Vanishing and exploding gradients
Gradients shrinking to nothing or growing without bound as they pass back through many layers is the failure that limited deep networks for years. Recognising the symptoms is what this item is for.
Xavier/Glorot and He initialization
Starting weights at the wrong scale causes exactly the failure above, and there are principled schemes for choosing that scale. They are defaults in every framework and worth understanding rather than trusting.
Dead units and saturation
Units that output zero for every input, and units pushed into the flat region of their activation, both stop learning permanently. Both are diagnosable by looking at activations.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.