6.1 From perceptron to multilayer network

What this is and why it exists

A neural network is one small arrangement repeated. Get that arrangement clear — on paper, with numbers you worked out yourself — and everything for the rest of this module is composition. Skip it, and every debugging session afterwards happens over a box you cannot see into, which is why this topic asks you to do a forward pass by hand before you touch a framework.

The vocabulary

• Unit — one weighted sum of inputs plus a bias, passed through a nonlinear function.
• Weight and bias — the numbers a unit learns: one multiplier per input, and one offset.
• Activation — the nonlinear function; also the number a unit outputs.
• Layer — many units side by side, all reading the same inputs.
• Forward pass — computing the output by working from inputs to prediction.
• Depth — how many layers are stacked.
• Width — how many units are in a layer.
• Universal approximation — the theorem that a single wide enough layer can approximate almost any continuous function.

The mental model

One unit is a weighted sum through a bend. Multiply each input by its own weight, add them up, add a bias, and pass the result through a nonlinear function. That is the whole thing. Without the nonlinearity the unit is a straight-line function of its inputs, and stacking straight-line functions produces another straight-line function — so a network of any depth without activations collapses to a single linear model. The nonlinearity is not decoration. It is the only reason depth means anything.

A layer is many units reading the same inputs. Each has its own weights, so each computes something different from the same information, and the layer's output is the list of what they all said. That list becomes the input to the next layer, which is the entire construction.

What depth buys is composition. The first layer sees the raw inputs and can only form simple combinations of them. The second layer sees those combinations and can form combinations of combinations. On images, the classic illustration is that early layers respond to edges, middle layers to shapes made of edges, and later layers to objects made of shapes. Nobody designed that hierarchy; it is what the arrangement produces when it is trained. Features that would have been hand-designed become something the network discovers, which is the single idea that separates this module from the classical one.

Then the theorem, and both halves of it. Universal approximation says that a network with one hidden layer, given enough units, can approximate almost any continuous function to any accuracy you like. That is a real result and it is quoted constantly. What it does not say is how many units "enough" is — the count can grow impossibly fast — nor how you would find the right weights, nor whether the resulting function would generalise to data it has not seen. It is an existence statement about representation, and training is a search problem, and generalisation is a third thing again. The theorem tells you a solution exists; practice is about finding one that also works on new data.

Which is why the field went deep rather than wide. For many functions, a deep network needs dramatically fewer units than a shallow one to represent the same thing, because it can reuse intermediate results rather than rebuilding them. A shallow network must construct every pattern from raw inputs; a deep one builds a vocabulary and then speaks in it. Width still matters — too narrow a layer is a bottleneck that throws information away — but between adding a layer and doubling a layer, depth is usually the more efficient purchase, and the architectures in this module are almost entirely arguments about how to make depth trainable.

Now do the ten-minute exercise, because it is the point of the topic. Take two inputs, a hidden layer of two units, one output. Invent small weights. Compute each hidden unit's weighted sum, apply the activation, write the number down. Feed those two numbers to the output unit, compute its sum, apply its activation, write down the prediction. That is a forward pass, complete, with nothing hidden. When a framework later reports a shape mismatch between layers, you will know exactly which multiplication it could not perform, because you have done that multiplication with a pen.

Two habits follow from having done it. Think of the weights between two layers as a rectangle of numbers, one row per input and one column per unit, and shape errors stop being mysterious. And remember that every unit in a layer sees the same inputs — so if you initialise all of them identically, they compute identical things, receive identical corrections, and stay identical forever. That failure has a name and a fix, and it is the subject of the initialisation topic.

What you should now be able to explain or do

Describe what one unit computes, naming each part. Say why a network without nonlinear activations is a linear model regardless of depth. Explain what a layer is and how layers feed one another. Give the composition argument for why depth produces hierarchies of features. State universal approximation accurately, including what it does not promise. Say why depth is usually a more efficient purchase than width. Work a forward pass through a tiny network by hand and read a shape mismatch afterwards.

Check yourself