5.15 Dimensionality reduction

What this is and why it exists

Dimensionality reduction compresses many features into a few, for two quite different purposes that people constantly confuse. One is a faithful summary you can feed to another model. The other is a picture you can look at. The methods that are good at the second are bad at the first, and reading a picture as though it were a summary is the most common error in this area — including in published work.

The vocabulary

• Principal component — a direction along which the data varies most, with each new one at right angles to those before.
• Variance explained — the share of the data's total spread a component accounts for.
• Whitening — rescaling components to equal variance after projecting.
• Embedding — a low-dimensional arrangement of points meant to preserve some notion of closeness.
• Perplexity — a setting controlling how many neighbours a point is arranged with respect to.
• Feature selection — keeping a subset of the original features.
• Feature extraction — building new features from combinations of them.

The mental model

PCA is a rotation followed by a truncation. Find the direction in which the data spreads most and call it the first component; find the direction of most remaining spread at right angles to it and call it the second; continue. Then keep the first few and discard the rest. Because the components are ordered by how much spread they capture, keeping the first few keeps most of the variation — and because the transformation is linear and reversible up to what you discarded, you can project new data the same way and even reconstruct an approximation.

Two consequences make it useful in practice. It is a genuine summary: the reduced features can go into another model, and the projection learned on training data can be applied unchanged to new data — which, as ever, means fitting it on the training portion only, inside the pipeline. And variance explained tells you how much you kept, so the number of components is a choice you can defend: plot the cumulative variance and take the number that reaches a level you can state.

Its limits follow from the same properties. It cares about variance, so a feature with a large range dominates unless you scale first — mandatory again. It is linear, so a relationship curled through the space is not captured. And the components are combinations of every original feature, which means they are usually not interpretable: "component one" is a weighted mixture of forty things, and naming it is storytelling unless the weights genuinely concentrate on a few features.

Whitening is a small addition worth knowing: after projecting, rescale each component to the same variance. That is helpful when the next model treats all directions alike, and harmful when the relative importance of the components was information you wanted to keep.

Then the visualisation methods, and the honest warning attached to them. They arrange points in two dimensions so that neighbours stay neighbours, which is a different objective from preserving structure. They are excellent for seeing whether classes separate at all, and for spotting an unexpected group. But the picture's geometry is largely an artefact. Cluster sizes are not meaningful; the distances between clusters are not meaningful; and the whole arrangement changes with the perplexity setting and with the random start. Two clusters drawn far apart may be no more different than two drawn adjacently.

So the rules for using them honestly are short. Run at several settings and only believe what survives all of them. Never quantify anything from the picture — no distances, no areas, no counts of visible groups presented as a finding. Never feed the two-dimensional coordinates into a downstream model as features. And when you present one, say what the settings were, because somebody reproducing it with different settings will get a different picture and should not think one of you is wrong.

The alternative method in the same family trades some of that away: it is faster on large data, tends to preserve more of the broad structure, and is more usable as an actual reduction rather than only a picture. It is not free of the same caveats, and the same rules apply, more gently.

Finally, selection against extraction, because they are answers to different requirements. Selection keeps original features, so the model stays explainable and the pipeline stays simple — you can still say "this uses attendance and prior score". Extraction builds combinations, which usually compresses better and costs you that sentence. If somebody will ask why the model said something, prefer selection. If the requirement is compression or speed, extraction is the stronger tool.

What you should now be able to explain or do

Describe PCA as a rotation and a truncation, and say what variance explained lets you defend. Say why scaling is mandatory and why the components are usually not interpretable. Say what whitening does and when it hurts. State the three things a two-dimensional embedding's geometry does not tell you. Give the four rules for using one honestly. Choose between selection and extraction from whether an explanation will be demanded.

Check yourself