4.22 Feedback, and the three network architectures

What this is and why it exists

Feedback is what separates a network that computes a function of its input from one that has a state. It is a small change to draw and a large change in what the object can do, and the single-loop system is the smallest place to see exactly what it adds. The three architectures that follow are not a taxonomy for its own sake. Each answers something the previous one cannot: a hidden layer exists because one layer of weights cannot represent certain structure at all, and a recurrent connection exists because a feedforward network has no memory of what it saw a moment ago.

The vocabulary

• Feedback — an output fed back to influence a later input.
• Unit delay — an element holding a value for one step before passing it on.
• Closed loop — a forward path and a return path forming a cycle.
• Layer — a group of units computing at the same stage.
• Feedforward — a network whose graph has no cycles.
• Single-layer network — one layer of computing units, fed by the inputs.
• Multilayer network — one or more hidden layers between input and output.
• Hidden layer — a computing layer that is neither input nor output.
• Fully connected — every unit feeds every unit in the next layer.
• Recurrent network — a network containing at least one loop.

The mental model

Start with one forward path and one return path. A signal enters, passes through the forward element, and part of the output is returned and combined with the input. The response of the whole is not the response of the forward element — it is the forward element divided by one minus the loop, which is the standard closed-loop result and the reason feedback can produce behaviour neither element has alone. Put a unit delay in the return path and the loop becomes memory. The output now depends on a weighted sum of everything that has entered, going backwards in time, with the weight falling as a power of the loop gain. Whether that sum settles or grows without bound depends on that gain, which is the stability question and the practical reason recurrent networks are harder to train.

Layers are counted where the arithmetic happens. A single-layer feedforward network is one layer of computing units fed directly by the inputs. The input layer is not counted, because nothing is computed there — it distributes values and does no work. This convention is universal and it is the source of a persistent confusion when someone counts the picture instead of the computation, so it is worth fixing now.

The multilayer network adds layers between input and output, and the hidden layer is what it is for. The usual description gives the sizes in order — so many inputs, so many hidden units, so many outputs. A hidden layer lets the network represent structure a single layer cannot. A single layer of weights followed by one function can only separate its input space in a limited way; the hidden units build intermediate features, and the output layer works on those instead of on the raw input. That is the whole reason the layer exists, and everything the next module does with depth is this idea repeated.

Connections may be left out on purpose. A fully connected network joins every unit in one layer to every unit in the next. A partially connected one does not, and the omission is a statement about the problem: these inputs are related to each other and those are not, this feature is local and does not need to see the whole image. Leaving connections out is where convolutional designs come from, and it is worth recognising here so that the later material reads as a specific choice rather than a new idea.

A recurrent network has at least one loop, and that gives it memory. Its output depends on what it has already seen as well as on what is arriving now, which is why sequences are handled here rather than by a feedforward design. The cost is that its behaviour over time can settle, oscillate or grow, and analysing which is the same stability question the single-loop system raised. This is the point where the electronics reading of the subject pays off directly: a network with feedback is a dynamical system, and the tools for asking whether a dynamical system settles are ones you already have.

What you should now be able to explain or do

Analyse a single-loop feedback system and write its closed-form response. Say what a unit delay in the return path turns the loop into, and what decides whether the resulting sum settles. Distinguish a single-layer feedforward network and explain why the input layer is not counted. Describe a multilayer network by its layer sizes and say exactly what the hidden layer buys. Distinguish fully from partially connected networks and give the reason for leaving connections out. Describe a recurrent network, say what the loop gives it, and name the cost.

Check yourself