6.18 Autoencoders and representation learning

What this is and why it exists

An autoencoder learns to reproduce its own input through a narrow middle, and the useful thing is never the reproduction — it is the narrow middle. Force information through a bottleneck and the network must decide what matters, which is a definition of representation learning that needs no labels at all. This topic covers that, the probabilistic version that makes the compressed space smooth enough to sample from, and the failure where the whole arrangement learns nothing while reconstructing perfectly.

The vocabulary

• Encoder — the part mapping an input to a compressed representation.
• Latent — the compressed representation itself; latent space — the space it lives in.
• Decoder — the part mapping a latent back to the input space.
• Bottleneck — the constraint that forces compression.
• Undercomplete — a latent smaller than the input.
• Denoising — training to reconstruct a clean input from a corrupted one.
• Reparameterisation — expressing a random sample so gradients can pass through it.
• Posterior collapse — the failure where the decoder ignores the latent entirely.

The mental model

The bottleneck is where the learning happens. Encode an input into fewer numbers than it started with, decode those numbers back, and score the result by how close the reconstruction is. To score well the encoder must keep whatever the decoder needs and discard the rest, so it is forced into a judgement about what is essential. Nobody labels anything; the data supervises itself.

The failure to understand first is the trivial one. If the bottleneck is not actually a constraint — the latent as large as the input, or a network with enough capacity to route values through — the autoencoder can learn something close to the identity function. Reconstruction is superb, the loss curve looks wonderful, and the representation is worthless because nothing was compressed. Reconstruction quality is not evidence that a representation is useful, and this is the single most important sentence in the topic. Judge the latent by what it can do, not by how well the decoder reproduced its input.

The variants each force better representations in a different way. Undercomplete is the plain version: the constraint is the size of the latent. Denoising autoencoders corrupt the input and ask for the clean version, which makes copying useless — you cannot copy what you were not given — and pushes the network to learn structure rather than storage. It is a strong default for that reason. Sparse autoencoders allow a large latent but penalise how many of its units are active at once, so the representation must be composed from a small number of pieces per input; this tends to produce parts-based representations, and it is the arrangement behind a good deal of current interpretability work on large models.

Then the question of what the latent space looks like. A plain autoencoder makes no promise about it. Points in between two training examples may decode to nothing meaningful, and there is no reliable way to pick a point that decodes to something plausible — so you can compress and reconstruct, but not generate. The test is interpolation: take two inputs, encode both, walk along the line between them, decode as you go. If the outputs change smoothly from one thing to the other, the space has structure; if they pass through garbage, it does not.

Variational autoencoders make the space well-behaved by changing what the encoder outputs. Instead of a point, it produces a distribution — a centre and a spread — and the latent is sampled from it. Two things follow. Training on samples rather than fixed points means nearby latents must decode to similar outputs, because the sampling keeps landing in the neighbourhood. And a second term in the loss pulls those distributions towards a standard reference distribution, so the latents fill a known region rather than scattering. Together these produce a space you can sample from: draw from the reference distribution, decode, and get something plausible.

The reparameterisation trick is the technical heart, and the problem it solves is precise. You cannot backpropagate through a random draw — there is no derivative of "sample from this distribution" with respect to the distribution's parameters. The trick is to move the randomness out of the path: draw a sample from a fixed standard distribution, then shift and scale it by the encoder's centre and spread. The result has exactly the distribution you wanted, and now the centre and spread are connected to the output by ordinary arithmetic that gradients pass through, while the random draw sits to one side as a constant input. Randomness becomes an input rather than an operation, and that is what makes the whole model trainable.

The variational form has its own characteristic failure. If the decoder is powerful enough to produce good outputs without consulting the latent, the term pulling the distributions towards the reference wins outright, the encoder outputs the reference distribution for every input, and the latent carries no information. The reconstruction may still look acceptable, and the representation is empty. The usual responses are to weight the two loss terms deliberately and to increase that weight gradually rather than applying it at full strength from the start.

Finally, the framing that matters in production: the embedding is the asset. A trained encoder turns your data into vectors that carry its structure, and those vectors outlive the task. They power similarity search, they become features for small supervised models trained on a handful of labels, they support clustering and deduplication, and they let you detect when new data no longer resembles the old. The representation is frequently more valuable than the model that produced it, and it is worth versioning and evaluating as a deliverable in its own right — which means evaluating it by a downstream task, such as training a simple classifier on top of it, rather than by reconstruction error.

What you should now be able to explain or do

Explain what the bottleneck forces and why no labels are needed. Recognise the trivial-identity failure and say why reconstruction quality proves nothing. Describe undercomplete, denoising and sparse variants and what each constrains. Test a latent space by interpolation and read the result. Say what a variational encoder outputs and how the two changes make the space samplable. Explain the reparameterisation trick and the exact problem it solves. Recognise posterior collapse and name the standard responses. Treat an embedding as a versioned asset and evaluate it downstream.

Check yourself