OE-11.3 Supervised Learning Networks

Work through these

• The perceptron network and the perceptron learning rule

  The first trainable network, and its learning rule. It is limited in a way the topic will make explicit, and understanding the limit is the point.

• Perceptron architecture and training algorithm

  Its structure and the training procedure written out. Working one example by hand is worth an hour here.

• ADALINE

  The linear version trained by minimising squared error, which connects this subject to ordinary regression. The connection is more useful than the name.

• MADALINE

  Several of the above combined, which gets past the single-unit limitation. It is a stepping stone to the general method below.

• The back propagation network and the BP learning rule

  The method that made deep networks trainable, and the centre of this whole course. Follow the chain rule through one small network by hand at least once.

• Input layer computation

  The first stage of that calculation, at the input. Splitting the derivation into three items is a kindness; treat them as one exercise.

• Hidden layer computation

  The middle stage, where the error is passed backwards. This is the part that gives the method its name and the part most often misunderstood.

• Output layer computation

  The last stage, where the error is first computed. It is where the loss function choice actually enters.

• Radial basis function networks

  A different architecture, where units respond to nearness rather than a weighted sum. It trains faster and generalises differently, which is the comparison to draw.

• Introduction to associative memory networks

  Networks that recall a whole pattern from part of it. It closes the supervised half and is the most brain-like idea in the course.