5.18 Hyperparameter optimization

What this is and why it exists

Tuning is where a great deal of time goes and where a surprising amount of it is wasted. This topic replaces poking at settings by feel with a procedure: choose a search method that suits the shape of the problem, give it a budget, and — most importantly — keep the search from quietly optimising against the very number you plan to report.

The vocabulary

• Hyperparameter — a setting chosen before training, as against a weight learned during it.
• Search space — the ranges and distributions the settings are drawn from.
• Grid search — every combination of a fixed list of values.
• Random search — combinations drawn at random from the ranges.
• Bayesian optimisation — using the results so far to decide what to try next.
• Successive halving — starting many candidates cheaply and giving more budget only to the survivors.
• Budget — the total compute you are willing to spend.
• Validation overfitting — a score inflated by having chosen against that validation data many times.

The mental model

Start with the observation that makes random search beat grid search, because it is not obvious and it is decisive. In any real problem, only a few hyperparameters matter much and the rest barely move the score. A grid over five values of four parameters spends 625 fits, but it only ever tries five distinct values of the one parameter that matters — the other 620 fits are re-testing those same five values with irrelevant variations. Random search with the same budget tries 625 distinct values of every parameter. When some parameters matter far more than others and you do not know which, random search explores the important dimension far better for the same cost.

Which points at the first practical step: define the space thoughtfully. Learning rates and regularisation strengths should be drawn on a log scale, because the meaningful difference is between 0.001 and 0.01 rather than between 0.501 and 0.51. Integer settings like depth need sensible bounds. And a range that turns out to have its best value at an edge is a range that was too narrow — widen it and search again, rather than accepting a boundary result.

Bayesian optimisation is the next step up. It builds a cheap model of how the score depends on the settings, then chooses the next trial to balance exploring uncertain regions against exploiting promising ones. It genuinely beats random search when each fit is expensive and the budget is modest — tens of trials rather than thousands — which is exactly the regime for a boosted model on a decent dataset. Its cost is a dependency and a little extra machinery, and its risk is over-trusting a small number of trials.

Successive halving is orthogonal and often the biggest practical win. Instead of training every candidate fully, train many for a small budget — few trees, few epochs, a fraction of the data — keep the best fraction, give them more, and repeat. Most candidates are eliminated cheaply, and the budget concentrates on the ones that look promising. It assumes that early performance predicts final performance, which is usually but not always true; a setting that starts slowly and finishes strongly can be cut early.

Then the failure this topic exists to prevent. Every time you choose against a validation set, you overfit it a little. Two hundred trials against the same folds and the best score is part real advantage and part the luck of that sample. The evidence you have picked up noise is familiar: the winner beats the runner-up by less than the spread across folds, and the ranking of the top candidates changes when you re-split.

Three defences, in increasing order of cost. Keep a final test set genuinely untouched until you have chosen, and report the number from it — which will be a little lower than your validation score, and that difference is the honest measure of how much you overfitted. Compare improvements against the fold-to-fold spread and treat anything smaller as a tie, preferring the simpler or faster model when it is one. And use nested cross-validation when the number must be defensible, since it puts the whole tuning procedure inside the evaluation.

Finally, budget the search deliberately rather than letting it expand. Tune what matters — a handful of settings, in the order the boosting topic gave, rather than everything at once. Stop when improvements fall below the noise. And remember the ordering that usually pays best: better features and more data beat better hyperparameters far more often than the reverse, and a day spent tuning a model that is missing an obvious feature is a day spent polishing the wrong thing.

What you should now be able to explain or do

Explain why random search beats grid search when some parameters matter more than others. Define a search space with sensible scales and say what a best-at-the-edge result means. Say when Bayesian optimisation is worth its extra machinery. Describe successive halving and the assumption it rests on. Explain validation overfitting and name the two symptoms. Give the three defences in order of cost. Say what usually beats hyperparameter tuning altogether.

Check yourself