4.21 The artificial neuron, and the network as a directed graph

What this is and why it exists

This is the structural half of neural networks, taught before any learning appears. It is deliberately in that order: what the object is, and only then how it is adapted. The description used here is the one from the signal-processing tradition rather than the software one, which is why it talks about signal-flow graphs, nodes and links instead of layers and tensors. If you have met block diagrams and transfer functions, this will feel familiar rather than new, and that familiarity is worth using. The biological story that opens the subject is a historical motivation, not a statement that the two things work alike.

The vocabulary

• Synapse — the junction where one cell influences another.
• Synaptic weight — the number multiplying one input.
• Bias — a constant added to the weighted sum.
• Induced local field — the weighted sum plus the bias, before the non-linearity.
• Activation function — the function applied to that field.
• Threshold function — output one above zero, zero below.
• Sigmoid function — a smooth S-shaped curve between zero and one.
• Signal-flow graph — directed links between nodes, each carrying a transfer.
• Synaptic link — a link that multiplies by a weight.
• Activation link — a link that applies the non-linear function.

The mental model

The biological picture, and how far it goes. A nerve cell collects signals through dendrites, combines them in the cell body, and sends a pulse down an axon that ends in synapses onto other cells. Some synapses encourage the receiving cell to fire and some discourage it. The cell fires when what it has received passes a threshold, and afterwards it cannot fire again for a short period. That description motivated the model historically and is where the analogy stops. The model keeps three ideas — many inputs, weights that can be positive or negative, and a threshold — and drops everything else. Treating the resemblance as more than that leads to bad arguments in both directions.

The comparison with a computer is worth having in mind. The brain is slow per element and enormously parallel, with an amount of connection no machine matches, and it stores what it knows in those connections rather than at addresses. It degrades gently: damage removes some capability rather than all of it. A conventional computer is the reverse on every count — fast, sequential, addressed storage, and no tolerance at all for a broken connection. These models were reaching for the first column and the reaching is what the subject is about.

A neuron is three operations. Multiply each input by its weight and add the results. Add a bias. Apply one function to the total. The sum before the function has a name worth knowing because it appears constantly: the induced local field. The bias is folded in as a weight on an input permanently fixed at one, which removes it as a special case and is the notation everything later assumes. When you meet a formula with an index starting at zero rather than one, that is what the zero term is.

Four activation functions, and the reason there are four. The threshold function outputs one above zero and zero below — the original, closest to the firing picture, and not differentiable, which will matter enormously in the next topic. The piecewise linear function rises straight through a middle region and saturates at both ends; widen its middle indefinitely and you have a linear amplifier, narrow it and you approach the threshold. The sigmoid is the smooth S-shaped curve between zero and one: strictly increasing, differentiable everywhere, with a slope you can set, and it approaches the threshold function as that slope grows. The tanh function is the same shape running between minus one and one, which centres the output on zero and is often better behaved for that reason. The smooth ones matter because a learning rule needs a derivative, and that single requirement is why the field moved off the threshold function.

A network is a directed graph, and three rules say how signals move in it. A signal flows along a link only in the direction its arrow points. A node's value is the sum of everything arriving at it, which covers the case of many links converging. The value at a node goes out unchanged along every link leaving it, independently of what those links do — so fan-out copies rather than divides. Two kinds of link exist and only two. A synaptic link multiplies the signal by a weight, which is linear. An activation link applies the non-linear function. Every neuron is a group of synaptic links converging on a node, followed by one activation link.

That gives four properties, and they define the object. Each neuron is a set of linear synaptic links, an externally applied bias represented as a link from an input fixed at one, and a possibly non-linear activation link. The synaptic links weight their inputs. The weighted sum defines the induced local field. The activation link squashes that field to produce the output. Anything satisfying those four is a neural network in this sense, which is a more useful definition than any drawing of circles and arrows.

What you should now be able to explain or do

Describe the biological neuron in its four parts and say precisely how far the analogy is meant to carry. Compare the brain and a computer on speed, size, storage and fault tolerance, and say what the comparison motivates. Write the neuron model with weights, bias, induced local field and output, and fold the bias in as a weight. Compare the four activation functions on smoothness, range and saturation, and say why differentiability decides which are used. Apply the three signal-flow rules and name the two kinds of link. State the four properties that make a directed graph a neural network.

Check yourself