4.21 The artificial neuron, and the network as a directed graph

Work through these

• Describe the biological neuron, and say how far the analogy is meant to go

  Dendrites, cell body, axon and synapse, and the fact that a cell fires when its inputs pass a threshold. The analogy motivated the model historically and is not a statement that the two work alike.

• Compare the brain and an artificial network on speed, size, storage and fault tolerance

  The brain is slow per element and massively parallel, stores in its connections and degrades gently; a computer is the reverse on every count. The comparison explains what these models were reaching for.

• Write the neuron model: weighted sum, bias, induced local field, output

  The sum of inputs times weights, the bias added to it, and the function applied to the result. Folding the bias in as a weight on a fixed input of one is the notation everything later uses.

• Compare the threshold, piecewise linear, sigmoid and tanh functions

  Four choices for the non-linear step, differing in whether they are smooth, where they saturate and what output range they give. The smooth ones matter because a learning rule needs a derivative.

• Apply the three signal-flow rules, and name the two kinds of link

  A signal moves along a link one way, a node adds everything arriving at it, and it sends the same value out along every outgoing link. Synaptic links multiply by a weight; activation links apply the non-linear function.

• State the four properties that make a directed graph a neural network

  Every neuron is a set of linear links plus a bias plus one possibly non-linear link, the links weight their inputs, the weighted sum is the induced local field, and the activation link squashes it.