5.10 Support vector machines and kernels

What this is and why it exists

Support vector machines are built on one clean idea — of all the boundaries that separate the classes, prefer the one with the most room on either side — and one clever one, which lets a linear method draw curved boundaries without ever computing the curved features. Both ideas outlived the algorithm's dominance and are worth having. The topic also explains, honestly, why boosted trees took the tabular crown.

The vocabulary

• Margin — the gap between the boundary and the closest points on either side.
• Support vector — one of those closest points; the only ones that determine the boundary.
• Hard margin — requiring perfect separation, which real data rarely allows.
• Soft margin — allowing some points to sit inside the margin or on the wrong side.
• C — how heavily those violations are penalised.
• Kernel — a function giving the similarity between two points, standing in for a dot product in a space you never build.
• RBF kernel — a similarity that falls off with distance, controlled by a width setting.
• Gamma — that width: large gamma means influence falls off quickly.

The mental model

Many lines separate two classes; the maximum-margin one is the line as far as possible from both. The appeal is intuitive — a boundary squeezed against the training points is precarious, and one with room either side survives a small change in the data. And it has a property worth noticing: only the closest points matter. Move a point far from the boundary and nothing changes; move a support vector and the boundary moves. The model is defined by a handful of examples, which makes it compact and also fragile in a specific way, since those few examples carry everything.

Real data is not cleanly separable, so the soft margin allows violations and charges for them. C is the price. A large C makes violations expensive, so the boundary contorts to classify the training data correctly — low bias, high variance. A small C tolerates violations, giving a wider, smoother margin that ignores some points — higher bias, lower variance. It is the regularisation dial from the earlier topic under another name, and it runs the other way round: large C means less regularisation, which is the one thing everybody gets backwards on first meeting.

Then the kernel trick, which is genuinely elegant. A linear method needs only the dot products between points, never the points themselves. So if you can compute what the dot product would be in some richer space, you can fit a linear boundary in that space without ever constructing it. A kernel is exactly such a function. The polynomial kernel corresponds to a space of feature products up to some degree; the RBF kernel corresponds to a space of infinitely many dimensions, which you could never build. The boundary is straight in that space and curved in yours, and no feature was ever expanded.

Two settings govern the RBF case and they interact. Gamma sets how far a single training point's influence reaches: large gamma means influence dies quickly, so the boundary can wiggle tightly around individual points, and the model tends to overfit. Small gamma gives a broad, smooth boundary. Combined with C, you are choosing on two axes at once, which is why the pair is nearly always tuned together on a grid.

Now the honest limitations. Scaling is mandatory — the RBF kernel is a function of distance, so unscaled features make it a function of your units, exactly as with nearest neighbours. And training cost grows steeply with the number of rows, because the method works with pairwise relationships between examples; somewhere in the tens of thousands of rows it stops being comfortable and well before a million it is impractical. That is the real reason boosted trees took over tabular work: they scale close to linearly in rows, handle mixed and unscaled features without ceremony, and usually match or beat a well-tuned kernel machine on the same data.

Where they still fit: small to medium datasets with many features — text with sparse high-dimensional vectors is the classic case, where a linear kernel is fast and strong — and problems where a clean margin genuinely exists. And the regression form applies the same idea inverted: instead of demanding points be outside a margin, it asks that they be inside a tube of a chosen width, and charges only for those outside it.

What you should now be able to explain or do

Say what the maximum-margin idea is and why only the closest points matter. Explain what C prices and which direction more C moves the bias-variance trade — including why the direction surprises people. Explain the kernel trick in terms of dot products, without expanding a feature space. Say what gamma controls and how it interacts with C. Give two reasons this is not the default for large tabular problems. Name where it still wins, and say what the regression form asks for instead of a margin.

Check yourself