OE-11.3 Supervised Learning Networks

What this is and why it exists

The perceptron has a famous limitation, and it is famous for a good reason: it nearly killed the field.

A single layer draws a straight boundary. So it cannot learn the exclusive-or function, whose two classes cannot be separated by any straight line. That result was published, taken as decisive, and funding fell away for years.

Backpropagation is the answer that revived it. And the thing to hold on to is that backpropagation is only the chain rule, applied carefully. Take the layers one at a time and it stops being intimidating.

The vocabulary

• Perceptron — a single layer with a threshold output.
• Decision boundary — the surface separating what the network calls one class from the other.
• Linearly separable — the classes can be divided by a straight boundary.
• Learning rule — the formula adjusting weights after each example.
• Hidden layer — a layer between input and output.
• Backpropagation — computing each weight's contribution to the error by the chain rule.
• Radial basis function — a unit responding to nearness to a centre rather than to a weighted sum.
• Associative memory — a network that recalls a whole pattern from part of it.

The mental model

The perceptron rule is direct. Present an example, and if the output is wrong, move the weights so this example is likelier to come out right. It converges when the classes are linearly separable, and only then. That condition is the whole story.

And exclusive-or is not linearly separable. Two of its four cases belong to one class and two to the other, arranged so that no straight line divides them. No amount of training fixes it, because the model cannot express the answer.

The adaptive linear variants are the step between. They adjust weights using the error before the threshold rather than after it, which gives a smoother rule. The multi-layer version shows that combining units produces boundaries a single one cannot.

Which is the resolution. A hidden layer lets the network compose boundaries, so shapes that no single straight line can separate become reachable. What was missing was not the architecture but a way to train it. How do you change a hidden weight when the error is only observed at the output?

Backpropagation supplies that. Compute the output, compare with what was wanted, then work backwards asking, at each layer, how much each weight contributed to the error. That is the chain rule from calculus, applied layer by layer. Take the input, hidden and output computations one at a time and write each down; the difficulty is bookkeeping, not concept.

Radial basis networks do something different worth noting. Their units respond to nearness to a centre rather than to a weighted sum. Each unit covers a local region rather than dividing the whole space. That makes them behave differently on inputs far from any training data, where they respond weakly rather than confidently extrapolating.

Associative memory closes the topic: networks that recall a complete stored pattern when given part of it, which is a different use again from classification.

What you should now be able to explain or do

Apply the perceptron rule and state the condition under which it converges. Explain why exclusive-or defeats a single layer, in terms of boundaries. Say what the adaptive linear variants changed. Say what a hidden layer makes possible and what was missing before backpropagation. Describe backpropagation as the chain rule applied layer by layer. Say how radial basis units differ and how that changes behaviour far from the data.

Check yourself