5.9 Naive Bayes

What this is and why it exists

Naive Bayes assumes something that is essentially never true — that every feature is independent of every other, given the class — and then works remarkably well anyway. That gap between a false assumption and a useful result is the most interesting thing in this topic, and understanding why it happens will change how you think about model assumptions generally.

The vocabulary

• Prior — how common each class is before looking at any features.
• Likelihood — how probable this feature value is within a class.
• Posterior — the probability of the class given the features, which is what you want.
• Conditional independence — the assumption that, once you know the class, the features tell you nothing about each other.
• Multinomial variant — for counts, which is the text case.
• Bernoulli variant — for presence or absence.
• Gaussian variant — for continuous features, assuming a bell curve per class.
• Smoothing — adding a small amount to every count so nothing has probability zero.

The mental model

Bayes' rule, in words: the probability of a class given the evidence is proportional to the probability of the class before seeing anything, times the probability of the evidence within that class. Score every class that way, take the largest, and that is your prediction.

The hard part is the probability of the evidence — a whole combination of feature values — which you have almost certainly never seen before in exactly that form. The naive assumption cuts that knot: assume the features are independent given the class, and the probability of the whole combination becomes the product of each feature's probability on its own. Each of those can be estimated by counting. That is the entire method, and it is why training is one pass over the data and prediction is a handful of multiplications.

In practice you add the logs rather than multiplying the probabilities, because multiplying a few thousand small numbers underflows to zero. Adding logs is the same comparison and it survives.

The assumption is false. In text, "New" and "York" are anything but independent; in medicine, symptoms co-occur. And yet the classifier is often good. The reason is worth internalising: you are not trying to estimate the probabilities correctly, you are trying to get the ranking of the classes right. The independence error inflates the evidence — the same signal counted several times through correlated features — but it usually inflates it in the same direction for the class that was already winning. The argmax survives; the probability does not. That is the topic's real lesson about assumptions: an assumption can be wrong in a way that does not touch the decision you are making with it, and knowing which part of your output is damaged is more useful than knowing that an assumption failed.

Which is exactly the trap. The predicted probabilities are badly calibrated — typically far too close to zero or one, because the repeated counting of correlated evidence piles up. So the labels are useful, the ranking is often useful, and the numbers should not be shown to anybody or used to set a cost-based threshold without calibrating them first, using the machinery from the metrics topic.

The three variants match three shapes of feature. Multinomial handles counts and is the standard for text, where the features are word occurrences. Bernoulli handles presence or absence and does better on short documents, where whether a word appears matters more than how often. Gaussian handles continuous features by assuming each is bell-shaped within each class — the weakest of the three assumptions in practice, and the reason Gaussian naive Bayes is more of a teaching example than a workhorse.

Smoothing is the small detail that makes it work at all. If a word never appeared in a class during training, its estimated probability is zero, and one zero in a product makes the whole score zero — so a single unseen word vetoes an otherwise obvious classification. The fix is to add a small constant to every count, so nothing is impossible, only unlikely. It is one line and without it the method is broken.

Where it still earns its place: a text classifier in ten lines and one pass over the data, which is a genuinely strong baseline for spam, topic labelling and sentiment; anywhere the training set is small, where its strong assumption acts as a form of regularisation; and anywhere you need something to compare the elaborate model against. The framing topic said to beat a baseline you could ship this afternoon — for text, this is that baseline.

What you should now be able to explain or do

State Bayes' rule in words and say which term the naive assumption makes tractable. Explain why the logs are added rather than the probabilities multiplied. Say why a false independence assumption still produces good labels, and exactly which part of the output it damages. Choose between the multinomial, Bernoulli and Gaussian variants from the feature type. Explain what smoothing prevents with the unseen-word example. Say when this is the right first thing to build.

Check yourself