PE1-6.4 Machine Learning: Supervised & Unsupervised

What this is and why it exists

There are many methods here and they are best learned by one question each. What shape of boundary does this draw?

Answer that for each and the catalogue becomes a set of tools with visible strengths, rather than a list of names to recall.

The vocabulary

• Supervised learning — learning from examples with known answers.
• Unsupervised learning — finding structure in data with no answers given.
• Overfitting — fitting the noise in the training data, so performance on new data is worse.
• Decision boundary — the surface separating one predicted class from another.
• Margin — the distance between a boundary and the nearest examples.
• Kernel — a transformation letting a straight boundary act as a curved one.
• Cluster — a group of similar examples, found without labels.
• Principal component — a direction along which the data varies most.
• Ensemble — many models combined into one prediction.

The mental model

Fitting a line is the simplest supervised method and the best place to see the central problem of the whole field. Fit a straight line and it may be too simple for the data. Allow a curve and it fits better. Allow enough curvature and it passes exactly through every training point. It is then worse on new data than the straight line was, because it has fitted the noise.

That is overfitting, and it is visible here in a way it is not elsewhere. Every method below has the same danger with a different name.

Now the boundaries.

Nearest-neighbour classification draws whatever the data says locally. There is no training step at all: the examples are the model. Its cost is at prediction time, when it must compare against the stored examples, which is the opposite of everything else here. It handles complicated boundaries naturally, and it suffers when there are many features, because in high dimensions everything is far from everything else.

A decision tree splits on one feature at a time, so its boundaries are aligned with the axes. That produces a model made of rectangular regions, and it produces something you can read. Explainability is its main advantage and it is a real one. You can print the path of tests that led to a decision, which matters wherever one must be justified.

Naive Bayes classifies with probabilities, assuming every feature is independent of every other given the class. That assumption is almost always false, and the method works anyway, which is worth understanding rather than accepting.

The reason is that classification does not need the probabilities to be right. It needs the ordering to be right. The independence assumption distorts the estimated probabilities, often badly, and it frequently distorts them in the same direction for every class. The most probable class is still the most probable one, and that is all the decision uses.

A support vector machine draws the boundary with the widest possible margin between the classes. The intuition is that a boundary squeezed against the examples is fragile. One sitting as far as possible from both sides is likelier to survive new data. The kernel trick then lets a straight boundary in a transformed space act as a curved one in the original space. The transformation is never computed. It was the strongest general method available before neural networks returned.

On the unsupervised side, two methods that reduce description length in different directions.

K-means groups the examples into a chosen number of clusters. Each centre moves to the mean of its members, members move to their nearest centre, and it repeats. It reduces many points to a few centres.

Principal component analysis finds the directions along which the data varies most and describes each example by its position along a few of them. It reduces many features to a few, and it is used constantly for visualisation and for making other methods workable.

Both share an honest difficulty. Each needs a number the data does not supply: how many clusters, or how many components. There are heuristics for choosing and none of them is definitive, and pretending otherwise is the failure to avoid.

Ensembles close the topic and are what usually wins on data of this kind. Bagging trains many models on different random samples and averages them. That reduces variance, because the individual errors are partly independent and partly cancel. Boosting trains models in sequence, each one concentrating on the examples its predecessors got wrong, which reduces bias.

The two combine differently for different reasons, and knowing which problem each addresses is more useful than knowing which usually scores higher.

What you should now be able to explain or do

Show overfitting on a fitted curve and name it in the other methods. Say what boundary each supervised method draws. Explain why naive Bayes works despite a false assumption. Say what a wide margin buys and what the kernel trick avoids computing. Name the number each unsupervised method needs, and say what bagging and boosting each reduce.

Check yourself