5.3 Logistic regression and classification

What this is and why it exists

Logistic regression is the workhorse classifier and the cleanest bridge from linear models to neural networks: a linear score squashed into a probability, trained by a loss that punishes confident mistakes. Done well it produces probabilities you can actually trust, which many more elaborate models do not. It also carries the most common misunderstanding in applied classification, which is about a number nearly everybody leaves at one half.

The vocabulary

• Linear score (logit) — the same weighted sum as linear regression, before squashing.
• Sigmoid — the function turning any number into one between 0 and 1: one divided by one plus e to the minus score.
• Odds — the probability of the event divided by the probability of it not happening.
• Log-odds — the natural log of the odds, which is exactly what the linear score is.
• Cross-entropy — the loss: minus the log of the probability the model gave to the true answer.
• Decision boundary — the set of points where the model is exactly undecided.
• Threshold — the probability above which you act.
• Softmax — the multi-class generalisation of the sigmoid.

The mental model

The model is linear regression with one extra step and one different loss. The linear score can be any number; the sigmoid maps it onto the range from zero to one so it can be read as a probability. Its shape is worth holding: at a score of zero the probability is one half, large positive scores approach one, large negative scores approach zero, and the curve is steepest in the middle — so the model is most sensitive to changes near the boundary and increasingly indifferent far from it.

The interpretation that makes coefficients readable is that the linear score is the log-odds. So a coefficient is the change in log-odds per unit of the feature, and exponentiating it gives the multiplier on the odds: a coefficient of about 0.69 multiplies the odds by two. That is the honest way to report one, and it carries the same caution as the previous topic — it is association within this model and this data, not an effect you can cause.

The loss is cross-entropy, and in words it is minus the log of the probability the model assigned to the correct answer, averaged over the rows. Read what that does: being right with high confidence costs almost nothing, being unsure costs a moderate amount, and being confidently wrong costs enormously, because the log of a tiny number is a large negative. That asymmetry is deliberate — it is what pushes the model towards honest probabilities rather than confident guesses. And its gradient has the same shape as the previous topic's, which is the bridge worth noticing: for each weight, the gradient is the average over rows of the feature times the difference between the predicted probability and the actual label. Predicted minus actual, times the feature. The same sentence as linear regression, with the residual now being a probability error.

The decision boundary is where the score is zero, which for a plain logistic model is a straight line, a plane, or its higher-dimensional equivalent. Curved boundaries come from basis expansion, exactly as in the previous topic.

Multi-class comes in two shapes. Softmax generalises the sigmoid: compute a score per class, exponentiate each, and divide by the total so they sum to one — one model, one training run, probabilities that are mutually exclusive. One-against-the-rest trains a separate binary model per class and normalises afterwards, which is simpler to reason about and gives probabilities that do not naturally sum to one. Softmax is the usual choice when the classes really are exclusive.

Then the part this topic exists for. The threshold is a business decision, not a property of the model. One half is an arbitrary default that most tools happen to use, and it is right only when a false positive and a false negative cost the same. For a screening test where a missed case is severe and a false alarm means a second look, the threshold belongs far below one half. For an action that annoys a customer, far above. The procedure is to name the cost of each kind of mistake, then choose the threshold that minimises the total expected cost — and to say plainly that this choice is being made, because if you do not make it deliberately, the tool has made it for you at 0.5.

Which is also why calibration matters here more than for most models. Choosing a threshold by cost only works if the probabilities mean something, and a well-calibrated classifier is one where, of the cases it scores near 0.8, about 80 percent really are positive. Logistic regression trained with cross-entropy tends to be well calibrated out of the box, which is a genuine and underrated advantage over models that produce scores rather than probabilities.

What you should now be able to explain or do

Describe the model as a linear score plus a squashing function, and say what the sigmoid's shape implies about sensitivity. Read a coefficient as a multiplier on the odds. State cross-entropy in words and say what it does to a confident mistake. Give the gradient in the predicted-minus-actual form and connect it to linear regression. Choose between softmax and one-against-the-rest. Set a threshold from the cost of each error and say why 0.5 is not a default worth keeping.

Check yourself