6.10 Debugging deep learning

What this is and why it exists

Deep learning bugs rarely crash. They converge to mediocrity, quietly, and look exactly like a problem that needs a bigger model. That is why debugging here is its own skill with its own procedure: a sequence of cheap tests that each rule out a whole class of fault, run in an order that finds the cause fastest. The instinct this topic is written against is adding layers or epochs to a bad curve, which buries the fault deeper and costs a day.

The vocabulary

• Silent bug — a defect that produces a running, converging, wrong model.
• Sanity check — a test whose expected result you already know.
• Overfitting one batch — training on a handful of examples until the loss reaches nearly zero.
• Range test — sweeping the learning rate to find where the loss falls fastest.
• Weight norm — the size of a layer's parameters, watched over time.
• Leakage — information reaching the model that will not exist at prediction time.

The mental model

Run the checks in this order, because each one rules out a class of fault and the early ones are the cheapest.

First: can the model memorise a single batch? Take eight examples, switch off augmentation, dropout and weight decay, and train on those eight repeatedly. A healthy setup drives the loss to nearly zero within a couple of hundred steps, because memorising eight examples requires no generalisation at all — it only requires that gradients reach the parameters and move them in the right direction. If it cannot, the bug is in your code, and no amount of tuning, data or capacity will help. This single test separates "my method is wrong" from "my program is wrong" in about a minute, and it is the most valuable habit in the topic. When it fails, look at: whether the loss is connected to the output you think it is, whether the optimiser was given the model's parameters, whether gradients are zeroed at the right point, whether the labels line up with the inputs, and whether an activation is saturating everything to a constant.

Second: is the loss where it should be at step zero? For balanced classification, the loss before any learning should be the negative logarithm of one over the number of classes — for ten classes, close to 2.3. A value far from that means the outputs, the targets or the loss are not what you think. It is a five-second check with a precise expected value, which makes it far stronger than looking at a curve.

Third: is the learning rate in range? Sweep it as described in the optimiser topic and read where the loss falls fastest. Most "the model will not learn" reports are a rate two orders of magnitude off, and the sweep resolves it in minutes rather than in an argument.

Fourth: what do the internals say? Log, per layer, the norm of the gradients, the mean and spread of the activations, and the norm of the weights, over the first few hundred steps. These are the instruments, and each has a signature. Gradient norms shrinking sharply as you go back through the layers means vanishing; growing means exploding, and clipping is the immediate response. Activations drifting towards zero or towards saturation locate the layer at fault. Weight norms growing without limit mean the decay is too weak or the rate too high. A ratio worth watching is the size of the update against the size of the weight — updates that are a vanishing fraction of the parameter are doing nothing, and updates comparable to the parameter itself are a run about to fail. Adding layers tells you nothing; these numbers tell you where.

Then the silent bugs, which are the ones that survive all of the above and still produce a mediocre model.

Shapes that broadcast instead of failing. A prediction of one shape compared against a target of a slightly different shape can broadcast into a valid loss that means nothing — the model trains, the loss falls, and the number being minimised is not the one you named. Print the shapes of the prediction and the target on the first batch, every time. This is the most common silent bug in the module.

Data that was never shuffled. If the file is ordered by class, unshuffled batches contain one class each, and the model learns to predict whatever it last saw. It converges, and the validation number is baffling. Check that shuffling is on for training, off for validation.

Normalisation applied inconsistently. Statistics computed on the training set must be reused at inference, not recomputed from whatever arrives. Recompute them and every prediction depends on the batch it came in — the same failure as leaving a normalisation layer in the wrong mode, arriving from the data side instead.

Labels misaligned with inputs. An off-by-one in an index, a sort applied to one array and not the other, a merge that reordered rows. The model then learns the best constant prediction it can and reports a plausible-looking loss. Look at a handful of examples with their labels rendered — actually look at the images, read the text — before believing any number.

Leakage from a column that knows the answer, which produces the opposite symptom: results that are too good. Treat an unexpectedly excellent validation score as a bug report rather than a success, and go and find out which feature is telling the model what it is meant to predict.

Two habits close the topic. Change one thing at a time and record what you changed and what happened, because a debugging session with three simultaneous changes produces no information. And keep a script that trains a tiny model on a tiny subset to convergence in under a minute — you will run it a hundred times, and every one of those runs is a question answered before you spend an hour on the real thing.

What you should now be able to explain or do

Run the checks in order and say what each rules out. Overfit a single batch and interpret failure as a code fault. Compute the expected initial loss for a balanced classification problem and check it. Use a range test rather than guessing the learning rate. Read gradient norms, activation statistics and update-to-weight ratios to locate a misbehaving layer. Name the four silent bugs and the check for each. Treat an unexpectedly excellent score as a bug report. Change one thing at a time and keep a fast subset script.

Check yourself