5.14 Clustering

What this is and why it exists

Clustering finds structure when nobody has labelled anything, which is most of the time. It is genuinely useful for segmentation, exploration and compression — and it is the technique most often reported as though it had discovered something, when in fact each algorithm imposes assumptions and will produce confident groups from data that has none. This topic is about choosing the algorithm for its assumptions, and about defending the number of groups you announce.

The vocabulary

• Centroid — the mean point of a cluster.
• k-means — assign each point to the nearest centroid, recompute the centroids, repeat.
• k-means++ — a smarter initialisation that spreads the starting centroids out.
• Inertia — the total squared distance from each point to its centroid.
• Hierarchical clustering — repeatedly merging the closest groups, producing a tree.
• Dendrogram — that tree, which you cut at a height of your choosing.
• Density-based clustering — groups defined by dense regions, with sparse points left unassigned.
• Silhouette — how much closer a point is to its own cluster than to the next nearest.

The mental model

Every clustering algorithm answers "what is a group" differently, and choosing one is choosing that answer.

k-means says a group is a round blob around a centre. It alternates two steps until nothing moves — assign each point to the nearest centre, then move each centre to the mean of its points — and it is fast on large data. Its assumptions are strong and they are usually unstated: clusters are roughly spherical, of similar size, of similar density, and separated by distance. It is also sensitive to where the centres start, which is what the improved initialisation addresses by spreading the first centres out; without it, a bad start produces a bad answer with no warning. And it needs scaled features, for the same reason nearest neighbours does.

Hierarchical clustering says a group is whatever is closest, repeatedly. It merges the two nearest groups over and over, building a tree from every point to one group, and you choose where to cut. Its advantage is that you get every number of clusters at once and can look at the structure before deciding; the dendrogram's shape often makes a natural cut obvious. Its cost grows quickly with the number of points, so it belongs to modest datasets.

Density-based clustering says a group is a dense region, and it has two properties the others lack. It finds clusters of any shape — crescents, rings, anything connected — because it grows outwards through dense neighbourhoods rather than assuming a centre. And it leaves sparse points unassigned as noise, rather than forcing every point into some group. In exchange, you set a neighbourhood radius and a minimum count instead of a number of clusters, and it does poorly when different regions have very different densities.

Then the number, which is the part you must be able to defend. Three tools, used together rather than separately.

The elbow plots total within-cluster distance against the number of clusters and looks for the bend where adding another stops helping much. It is a useful first look and it is frequently ambiguous — real data often has no clear elbow, and choosing one anyway is where confident nonsense begins.

The silhouette measures, for each point, how much closer it is to its own cluster than to the nearest other one, averaged over all points. Higher is better, and unlike the elbow it has an interpretable scale: values near zero mean the point sits on a boundary and the grouping is not saying much.

Stability is the most convincing and the least used. Cluster repeated samples of your data — or the same data from different starting points — and see whether you get the same grouping. A structure that survives resampling is likely to be real; one that changes each time is an artefact of the algorithm, however good its silhouette. If you report a number of segments, report the stability check too.

Then the trap, stated plainly. k-means will carve uniform noise into k confident clusters and report inertia for each. Nothing in the output says "there was no structure here" — the algorithm's job is to partition, and partition it will. So the last step before showing anybody a segmentation is to ask whether the same procedure on shuffled or synthetic random data of the same shape produces something that looks equally convincing. If it does, you have described the algorithm, not the customers.

And a practical closing note: clustering results become useful when the groups get names and descriptions. A segment is worth something when somebody can say "these are the students who start strong and stop after three weeks" — which means profiling each cluster on features you did not cluster on, and checking that the description holds.

What you should now be able to explain or do

State what each of the three algorithms assumes a group is, and choose between them for described data. Say what k-means requires of its features and why the initialisation matters. Explain what a dendrogram gives you that a fixed k does not. Name the two properties density-based clustering has that the others lack. Use the elbow, the silhouette and a stability check together, and say why stability is the strongest. Describe the noise test, and say what it protects you from claiming.

Check yourself