PE1-6.3 Neural Networks

What this is and why it exists

This topic is the vocabulary layer. It says what these systems are made of and how they are described, and the next topic does the algorithms.

One framing makes it useful rather than decorative. A network is a directed graph: nodes compute, edges carry weights, and the arrangement is a design choice rather than a diagram to memorise.

The vocabulary

• Neuron — one computing unit, combining its inputs and applying a function to the result.
• Weight — the number scaling one input to a unit.
• Activation function — the function applied to a unit's combined input.
• Directed graph — nodes joined by edges with a direction.
• Feedforward network — one whose edges run only towards the output, with no loops.
• Feedback network — one with loops, so a unit's output can influence its own later input.
• Learning rule — the procedure that changes the weights.
• Supervised learning — learning from examples labelled with the wanted answer.
• Unsupervised learning — learning structure from unlabelled data.
• Reinforcement learning — learning from a reward signal rather than from labelled answers.

The mental model

The biological analogy is the usual motivation and it is loose. Real neurons are far more complex, they are not arranged like this, and the learning that happens in a brain is not what happens here. Treating the analogy as loose is what keeps expectations honest, and it costs nothing, because the mathematics does not depend on the analogy being close.

A unit is simple. It scales each input by a weight, adds them, and applies a function to the total. Without that function the whole network would collapse to a single linear operation however many layers it had, so the function is not a detail. It is what makes depth mean anything.

The graph framing is what makes architecture a design choice. Nodes compute. Edges carry weights and a direction. Once a network is drawn that way, architecture becomes a question about the graph. How many nodes, in how many layers, connected how densely, with which edges present.

The presence or absence of loops is the deepest structural distinction.

A feedforward network has no loops. Signals travel one way, the output depends only on the current input, and the network has no memory. It is a function from input to output, and training it is comparatively well behaved.

A feedback network has loops, so a unit's output can influence its own later input. That gives the network state, which means it can respond to a sequence rather than to a single input. It also makes it harder to train and harder to reason about, because the network's behaviour now depends on where it has been.

Learning is where the weights change, and the taxonomy matters more than the biology. Two questions separate the kinds. What signal says the weights should change, and where does that signal come from?

In supervised learning the data carries the wanted answer. The signal is the difference between what the network produced and what it should have. In unsupervised learning there is no wanted answer, and the signal comes from structure in the data itself, such as similarity between examples. In reinforcement learning there is neither a labelled answer nor structure. A reward arrives after a sequence of actions, so assigning credit to any single decision is the central difficulty.

The rules themselves are largely variations on one move. Adjust each weight in the direction that reduces the error, by an amount related to its contribution.

Learning tasks are what these systems are asked to do, and naming them connects the machinery to the applications in the final topic. Classification, putting an input into one of a set of categories. Regression, producing a number. Pattern association, recalling a stored output from a partial or noisy input. Clustering, grouping without labels. And control, producing actions that move a system towards a wanted state.

What you should now be able to explain or do

Say why the biological analogy is loose and why that costs nothing. Explain what the activation function is for and what happens without it. Read a network as a directed graph and state architecture questions in those terms. Say what feedback gives a network and what it costs. Separate the kinds of learning by what signal changes the weights.

Check yourself