4.22 Feedback, and the three network architectures

Work through these

• Analyse a single-loop feedback system, and read its closed-form response

  One forward path and one return path, giving a response that depends on the loop as a whole rather than on either part alone. The unit delay in the return path is what turns the loop into memory.

• Distinguish a single-layer feedforward network, and say why the input layer is not counted

  One layer of computing units fed directly by the inputs, with no arithmetic happening at the input side. Counting layers by where the arithmetic happens is the convention every later description assumes.

• Describe a multilayer feedforward network, and say what a hidden layer buys

  One or more layers between input and output, described by the count of units in each. The hidden layer is what lets the network represent structure that no single layer of weights can.

• Distinguish fully connected from partially connected networks

  Whether every unit in one layer feeds every unit in the next, or only some of them. Leaving connections out is a deliberate statement about the problem, and it is where convolutional designs come from.

• Describe a recurrent network, and say what the feedback gives it

  A network with at least one loop, so its output depends on what it saw earlier as well as on its input now. That memory is the reason sequences are handled here and not by a feedforward design.