6.3 Activations and initialization

What this is and why it exists

A network that flatlines at epoch three is usually not badly designed and not short of data. It has a gradient problem, and gradient problems are decided by two choices made before training started: which nonlinearity each layer uses, and what scale the weights began at. This topic is those two choices, the failure each prevents, and how to tell from the inside which one you are looking at.

The vocabulary

• Saturation — an activation pushed into a flat region, where its derivative is nearly zero.
• Vanishing gradient — a gradient that shrinks towards nothing as it travels back through depth.
• Exploding gradient — the same trip, multiplying up instead of down.
• Dead unit — a unit that outputs zero for every input it will ever see, and can no longer recover.
• Fan-in and fan-out — how many inputs a unit has, and how many places its output goes.
• Symmetry breaking — starting units differently so they can learn different things.

The mental model

Gradients travel by multiplication, and that is the whole story. The backward pass multiplies a local derivative at every layer it passes through. Multiply numbers slightly below one, twenty times, and you have almost nothing left — the early layers receive a correction too small to move them, and they stop learning while the last few layers carry on. Multiply numbers slightly above one and you get the mirror image: gradients grow until the update overshoots and the loss becomes meaningless. The whole business of activations and initialisation is keeping those per-layer factors near one.

The activations, and what distinguishes them. The older saturating functions squash their input into a fixed range, which sounds harmless and is the direct cause of vanishing gradients: once an input is large in either direction, the output is flat and its derivative is nearly zero, so nothing passes back. tanh is the better-behaved of the classic pair because it is centred on zero, and it still saturates.

The rectified family fixed this by refusing to saturate on the positive side: negative inputs give zero, positive inputs pass through unchanged, and the derivative for positive inputs is exactly one. Gradients survive depth, and the function costs a comparison. Its own failure is on the other side — a unit whose inputs are always negative outputs zero always, receives zero gradient always, and never recovers. That is a dead unit, and a layer where most units are dead is a layer that has quietly stopped existing.

The leaky variant addresses that by giving negative inputs a small slope instead of zero, so a gradient still trickles through and the unit can come back. The smooth modern variants bend gently around zero rather than turning a corner, and they tend to train slightly better in deep stacks — most visibly in transformers, which is where you will meet them. The honest summary: the choice affects training speed and stability much more than final accuracy. Use a rectified variant by default, use the smooth ones where the architecture you are copying uses them, and do not spend a week on it.

Initialisation is the other half, and it is the same arithmetic seen from the start. Weights drawn too large make each layer amplify its input, activations grow with depth, and the forward pass overflows or saturates. Too small, and the signal fades to nothing before it reaches the end. The principled schemes fix the scale so that the variance of the signal is roughly preserved from layer to layer — one scheme derived by considering both the forward and backward passes for symmetric activations, and a second derived specifically for the rectified family, which is larger by a factor that compensates for half the outputs being zeroed. Match the scheme to the activation: the rectified-family scheme with rectified units, the symmetric one with symmetric activations. Frameworks do this by default and the default is usually right — the value of understanding it is that when a custom layer trains strangely, initialisation is on the short list.

Then the failure that has nothing to do with scale. Initialise every weight in a layer to zero, or to the same constant, and every unit in that layer computes the same output, receives the same gradient, and applies the same update. They remain identical forever. A layer of five hundred identical units has the capacity of one, and the network is secretly a much smaller and much dumber model that reports no error at all. Randomness in initialisation is not for luck; it is what makes units able to differ. Biases may safely start at zero — the weights are what need breaking apart.

Diagnosing is easier than remembering. During the first hundred steps, print the mean and spread of each layer's activations and the norm of each layer's gradient. Activations shrinking towards zero with depth, or gradient norms an order of magnitude smaller in the early layers, is vanishing. Growing activations, or a loss that becomes meaningless in one step, is exploding. A rectified layer where the fraction of units outputting zero is climbing towards all of them is dying, and the usual cause is a learning rate large enough to have driven them there. Those three printouts turn "it stopped learning" into a specific, fixable statement.

What you should now be able to explain or do

Explain vanishing and exploding gradients as repeated multiplication through depth. Say what saturation is and why it stops learning. Describe the rectified family and its dead-unit failure, and what the leaky variant changes. Choose an activation without agonising, and say what the choice actually affects. Explain what the initialisation schemes are preserving and why one is scaled larger than the other. Say why all-zero or all-equal initialisation is fatal and what randomness is really for. Read activations and gradient norms to name which failure you have.

Check yourself