OE-11.2 Essentials of ANNs & Learning Strategies

What this is and why it exists

One sentence in this unit is worth more than the rest, so here it is first. Without a non-linearity, any depth collapses to a single layer.

Stack linear operations and the result is another linear operation. A hundred layers of pure weighted sums compute exactly what one layer could. The activation function is what makes a network more than stacked linear algebra, and that is why it exists at all.

The learning strategies then divide by one question: what feedback does the network receive?

The vocabulary

• Activation function — the non-linear step applied to the weighted sum.
• Step function — outputs one of two values. Simple, and not differentiable.
• Sigmoid — a smooth S-shaped function, differentiable everywhere.
• Rectified linear — zero below zero, and the input itself above.
• Architecture — how many layers and how they connect.
• Supervised learning — the wanted output is supplied with each example.
• Unsupervised learning — no wanted output; structure must be found.
• Reinforcement learning — only a reward, arriving later.
• Learning rule — the formula that changes the weights.

The mental model

Start with why the activation function is not optional. A layer computes a weighted sum, which is a linear operation. Feed one linear operation into another and you get a linear operation. So depth without non-linearity buys nothing whatsoever — the network is a single layer wearing extra parameters.

Insert a non-linear function after each layer and that collapse stops. Depth now builds representations that a single layer cannot express, which is the entire reason to have layers.

The choices then differ in a property that matters for training. A step function is simple and has a derivative of zero almost everywhere, which gives gradient methods nothing to work with. A sigmoid is smooth and differentiable, which is what made gradient training possible. The cost is flattening at its extremes, where the gradient becomes very small and learning stalls. The rectified variety is cheap and does not flatten on the positive side, which is why it displaced the sigmoid in deep networks.

Architectures divide by connectivity, and that reconnects to the previous topic: forward-only, or with connections that lead back.

The learning strategies divide by feedback. Supervised: each example carries the wanted answer, so the error is available directly. Unsupervised: no wanted answer, so the network must find structure — grouping, or a compressed representation. Reinforcement: neither answer nor structure, only a reward arriving after a sequence of actions, which makes assigning credit to any single decision the central difficulty.

The learning rules themselves are largely variations on one move. Adjust each weight in the direction that reduces the error, by an amount related to its contribution. Working a numerical weight update by hand, once, does more for understanding than reading three of them.

What you should now be able to explain or do

State why depth without non-linearity collapses to one layer. Compare activation functions by differentiability and by where they flatten. Say why the sigmoid enabled gradient training and why it was later displaced. Classify a network by connectivity. Distinguish the three learning strategies by the feedback each receives. Say what makes credit assignment hard in the third. Work a weight update by hand.

Check yourself