6.4 Loss functions

What this is and why it exists

The loss is the only thing training actually optimises. Everything else — the architecture, the optimiser, the schedule — is machinery for reducing whatever number you named, so naming the wrong number produces a model that is excellent at the wrong thing and reports no fault. This topic is how to choose a loss by what it penalises rather than by what the tutorial used, and the numerical care that keeps the chosen loss from failing silently.

The vocabulary

• Loss — the number training minimises, computed per example and averaged.
• Outlier — an example whose error is large and whose largeness is not meaningful.
• Hard example — one the model currently gets wrong, or is unsure about.
• Overconfidence — predicted probabilities closer to zero and one than the evidence supports.
• Margin — how far apart a loss insists two things should be pushed.
• Numerical stability — whether a formula stays accurate in finite-precision arithmetic.
• Fused operation — one operation implementing several steps together, for stability or speed.

The mental model

A loss is a statement about what mistakes cost you. Ask that question first, in the language of the problem, and the choice usually makes itself.

For regression, the three standard losses differ only in how harshly they punish being far off. Squared error grows with the square of the mistake, so one example that is ten units out matters as much as a hundred examples that are one unit out — appropriate when a large error is an emergency, and destructive when it is a typo in the data. Absolute error grows in proportion, so a wild value is one more example rather than the whole gradient, which is what you want when outliers are noise. The Huber loss is the compromise built for exactly this: squared for small errors, proportional for large ones, with a threshold you set for where "large" starts. Choose by asking whether your large errors are outliers or emergencies — if the data contains recording mistakes, the proportional behaviour is right; if being far wrong is genuinely dangerous, the squared behaviour is right.

For classification, cross-entropy is the default and the two variants are answers to observed problems. Cross-entropy charges you the negative logarithm of the probability you assigned to the correct answer, so being confidently wrong is enormously expensive and being right without conviction is mildly expensive. That is the right shape for probabilities, and it is why this loss also gives calibrated outputs when nothing interferes with it.

Focal loss addresses a specific situation: a mass of already-solved examples whose small individual losses add up to more gradient than the handful of hard ones. It multiplies each example's loss by a factor that shrinks as the model becomes confident about it, so the settled majority quietly fades and the hard minority dominates. It was designed for dense detection, where background overwhelms foreground, and that is still where it earns its place — reach for it when the training signal is being drowned by examples the model already handles.

Label smoothing addresses the opposite tendency. Cross-entropy against a target of exactly one pushes the model to be ever more certain, and there is no point at which it is satisfied; the result is overconfidence and worse calibration. Smoothing replaces the target with something slightly less than one and spreads the remainder over the other classes, which tells the model to be confident but not infinitely so. It usually costs a little accuracy and buys better-behaved probabilities, and it is standard in large-scale image and language training.

The contrastive family learns representations rather than answers. Instead of a label, you supply pairs: these two things are similar, those two are not. A contrastive loss pulls the similar pair's representations together and pushes dissimilar ones apart until they are at least a set margin away — beyond that margin it stops caring, which is the detail that keeps it from pushing forever. The triplet form makes the comparison relative: an anchor, something similar and something different, with the requirement that the similar one be closer than the different one by that margin. The practical difficulty in both is choosing which negatives to compare against, because a randomly chosen negative is usually already far away and teaches nothing. These losses are the mechanism behind most of the self-supervised work later in this module, and they are how you learn a useful embedding when nobody has labelled anything.

Finally, numerical stability, which is where a correct loss becomes a wrong number. The natural way to write cross-entropy is to compute a softmax and then take its logarithm. Both halves are unsafe: exponentials of moderately large scores overflow to infinity, and probabilities that round to zero produce the logarithm of zero, which is not a number, and that value propagates through every gradient in the batch until the whole model is filled with them. The stable formulation subtracts the largest score before exponentiating — which changes nothing mathematically and everything numerically — and combines the logarithm with the softmax so the exponential is never formed on its own. This is why frameworks offer a fused cross-entropy that takes raw scores rather than probabilities: the fused version is the stable one, and the separate version exists to be composed with other things by people who know what they are doing. Write a custom loss and you inherit the whole problem, so clamp before dividing, add a small constant inside logarithms, and test the loss on extreme inputs before trusting it in a run.

What you should now be able to explain or do

State what a loss is choosing on your behalf. Pick between squared, absolute and Huber by asking whether large errors are outliers or emergencies. Explain the shape of cross-entropy and why it suits probabilities. Say what problem focal loss and label smoothing each solve, and when to reach for them. Describe contrastive and triplet losses, the role of the margin, and the difficulty of choosing negatives. Explain why computing softmax and then a logarithm is unsafe, and what the fused operation does about it. Write a custom loss with the standard stability precautions.

Check yourself