6.12 Convolutional networks

What this is and why it exists

A convolution is the previous topic's sliding filter, with the weights learned instead of designed. That one change is the whole of this topic, and it works because it builds in two assumptions that happen to be true of images and of signals: what matters is local, and a pattern means the same thing wherever it appears. This topic is those assumptions, the four settings that describe any convolution, and the shape arithmetic that produces every beginner's first convolutional network bug.

The vocabulary

• Kernel — the small array of learned weights; also called a filter.
• Stride — how far the kernel moves between positions.
• Padding — values added around the edges so the output keeps a chosen size.
• Dilation — spacing inserted between the kernel's points, widening its reach.
• Feature map — the output of one kernel applied across the input.
• Pooling — reducing a region to a single value, shrinking the map.
• Receptive field — the region of the input one output value depends on.
• Parameter sharing — using the same kernel at every position.

The mental model

Two priors do all the work. Locality: a pixel's meaning is decided by its neighbours, not by pixels across the image, so a unit only needs to look at a small window. Translation equivariance: an edge is an edge wherever it occurs, so the same weights should apply everywhere. Together these replace a fully connected layer's enormous parameter count with a handful of numbers reused at every position. A layer connecting two modest images densely needs millions of weights; a kernel across the same image needs a few dozen. Parameter sharing is why these networks are trainable on ordinary hardware at all, and it is also a form of regularisation, because a shared weight must work everywhere rather than memorising one location.

Four settings describe any convolution, and PyTorch defines each precisely. Stride "controls the stride for the cross-correlation" — how far the window moves between positions, so a stride of two halves the output size in each dimension. Padding "controls the amount of padding applied to the input", either a number of pixels or one of two named modes, and it decides whether the output shrinks at every layer or keeps its size. Dilation "controls the spacing between the kernel points", which widens what the kernel covers without adding a single weight — the cheapest way to see more context, and the standard tool in segmentation. Groups "controls the connections between inputs and outputs", with the input and output channel counts both divisible by the group count; taken to the limit, each channel is filtered independently, which is the depthwise convolution that the efficient architectures are built from.

Then the arithmetic that will bite you. The output size along a dimension is: take the input size, add the padding on both sides, subtract the effective kernel size, divide by the stride, discard the remainder, and add one. The effective kernel size is where dilation enters — a three-wide kernel with a spacing of two covers five positions, not three. Two consequences are worth memorising. An odd kernel with padding of half its size, at stride one, leaves the size unchanged, which is why odd kernel sizes are the convention. And discarding the remainder means information at the edge can be dropped silently when the stride does not divide evenly, which is a real and quiet asymmetry.

Channels are the dimension people forget. A kernel spans every input channel at once: a three-by-three kernel over sixty-four input channels is not nine weights but nine times sixty-four, plus a bias. The number of kernels in the layer is the number of output channels. Say the shape out loud when defining a layer — input channels, output channels, kernel size — and the parameter count stops being a surprise.

Pooling and the receptive field go together. Pooling reduces each small region to one value, usually its maximum, which shrinks the map, discards precise position, and keeps the strongest response. Its purpose is to let later layers see more of the image for the same cost. The receptive field is the region of the original input that one output value depends on, and it grows as you stack layers: each convolution adds the kernel's reach, and each stride or pooling step multiplies what follows. A network cannot use evidence outside its receptive field, so if the object of interest is larger than the receptive field at the layer where the decision is made, no amount of training will fix it — the answer is more depth, a larger stride early, or dilation. Compute it when a model inexplicably fails on large objects. Modern designs frequently replace pooling with strided convolutions, which downsample with learned weights instead of a fixed rule.

The classic architectures read as one argument, and reading them in order is the fastest way to understand modern design. The earliest practical network established the pattern that everything since has followed: alternating convolution and pooling to reduce the spatial size while increasing the channel count, ending in a dense classifier. The network that made the field pay attention scaled that up onto accelerators and added rectified activations and dropout. The one after it made a clean argument that stuck: use only small kernels, stacked, because two three-wide layers cover the same reach as one five-wide layer with fewer parameters and an extra nonlinearity in between. Small kernels stacked deep is the lesson, and it is why you rarely see a large kernel today.

The same operation along one dimension works on signals and time series, and this is the bridge to the electronics side of the curriculum. A one-dimensional convolution over a waveform learns filters exactly as the classical topic designed them by hand, dilation stacks reach far back in time cheaply, and the priors still hold: a pattern in a signal is local, and it means the same thing wherever it occurs. If you have worked with filters and impulse responses before, this is that material with the coefficients learned from data.

What you should now be able to explain or do

State the two priors and say what each buys. Explain parameter sharing and why it is both efficient and regularising. Define stride, padding, dilation and groups, and say what each controls. Compute an output size, including with dilation, and say what the discarded remainder does. Count the parameters of a convolutional layer correctly, including channels. Explain pooling and compute a receptive field, and recognise the failure it explains. Give the argument for small stacked kernels. Apply the same operation to a one-dimensional signal.

Check yourself