6.13 Modern vision architectures

What this is and why it exists

Modern vision is a menu with a budget axis, and choosing from it well is a more useful skill than knowing any one entry in detail. This topic is the four families you will actually meet, the single architectural idea that made all of them possible, and the working rule that keeps people from defaulting to the largest model available and paying for it in every dimension at once.

The vocabulary

• Residual connection — a path that carries a block's input forward and adds it to the block's output.
• Skip connection — the general term for any connection that bypasses layers.
• Bottleneck block — a block that narrows the channels, does the expensive operation, then widens again.
• Depthwise separable convolution — filtering each channel independently, then mixing channels with a one-by-one convolution.
• Compound scaling — growing depth, width and input resolution together in fixed proportion.
• Patch embedding — cutting an image into fixed squares and treating each as a token.
• Inductive bias — an assumption built into an architecture, which reduces how much data it needs.

The mental model

One idea underlies everything here: the residual connection. Before it, stacking more layers past a few dozen made networks worse, and not because of overfitting — the training error rose too, which meant the deeper network could not even learn what the shallower one had. The diagnosis was optimisation: it is hard for a stack of layers to learn to pass its input through unchanged, and yet that is what the extra layers ought to do if they have nothing to add.

The fix is to add a path that carries the input forward and adds it to the block's output, so the block only has to learn the difference it wants to make. Doing nothing then costs nothing — the block outputs zero and the input passes through. Gradients get a route back to the early layers that does not pass through every intervening transformation, which is the vanishing-gradient problem solved structurally rather than by tuning. Depth became a setting rather than a struggle, and every architecture in this topic, convolutional or otherwise, uses residual connections.

The residual convolutional family is still the reliable default. Deep stacks of small kernels, bottleneck blocks so that depth stays affordable, batch normalisation throughout, sizes from small to large so you can pick by budget. Pretrained weights are available everywhere. On a few thousand images with a normal compute budget, this is the answer that works, and starting anywhere else needs a reason.

The efficient families are the same construction optimised for a budget rather than a leaderboard. Their central device is the depthwise separable convolution: filter each channel on its own, then use a one-by-one convolution to mix channels. A standard convolution does both at once and pays for the product; splitting them costs a fraction of the operations for very nearly the same result. Add an inverted bottleneck — widen, filter cheaply, narrow again — and you have the block that phone-scale models are built from. The other contribution is compound scaling: rather than deepening a network arbitrarily, grow depth, width and input resolution together in fixed proportion, on the argument that a deeper network that cannot see more detail is unbalanced. Choose by budget rather than by leaderboard position, and state the budget first: the memory available, the latency permitted, and whether it runs on a phone, a server or a microcontroller.

Vision transformers brought the two halves of the field together. Cut the image into fixed patches, flatten each into a vector, add a position marker, and feed the sequence to the attention architecture that the next few topics cover. Attention lets any patch attend to any other from the first layer, so there is no receptive-field limit to work around.

What they give up is the assumptions. A convolution builds in locality and translation equivariance for free; attention over patches assumes nothing and must learn both from data. That is the whole trade: with enough data — and it is a lot — the model learns better assumptions than the ones we imposed, and below that threshold it is worse than a convolutional network that was told the answer in advance. On a modest dataset a pretrained residual network frequently beats a transformer trained from scratch, at a fraction of the cost, and reaching for the largest available model is the most common expensive mistake in this topic. The qualifier that matters is "from scratch": a transformer pretrained on a large corpus and fine-tuned on your few thousand images is a different proposition, and often an excellent one.

Then the convergence, which is the point of the fourth family. When people took the residual convolutional design and applied the incidental lessons from transformers — larger kernels, fewer activations, layer normalisation instead of batch normalisation, a different block ordering, modern training recipes — the resulting convolutional network performed comparably to the transformers it was chasing. The lesson people drew is the useful one: much of the gap attributed to attention was actually the training recipe, and architecture matters less than the combination of data, augmentation, schedule and scale. Read a comparison between two architectures by asking first whether they were trained the same way.

The working procedure, then. Write down the constraints before opening a model zoo: how much labelled data, what latency, what memory, what hardware. Start from a pretrained mid-sized residual network, because it is the strongest default per unit of effort. Change architecture only when a constraint forces it — an efficient family for a phone, a transformer when you have data at scale or a strong pretrained starting point. And measure the baseline before shopping, because the difference between two architectures is routinely smaller than the difference made by better augmentation.

What you should now be able to explain or do

Explain what problem residual connections solved and why the fix is structural. Say what a bottleneck block is for. Describe depthwise separable convolutions and why they cost less. State what compound scaling balances. Explain patch embedding and what a vision transformer gives up along with what it gains. Say when a pretrained convolutional network beats a transformer and what the qualifier is. State the lesson of the converged convolutional design. Choose an architecture from stated constraints rather than from a leaderboard.

Check yourself