5.13 Gradient boosting: XGBoost, LightGBM, CatBoost
You can win most tabular problems you meet.
Before:03. Data Handling & AnalysisUnlocks:06. Deep Learning10. Production & MLOps13. Capstones, Portfolio & Interviews
Gradient boosting fits trees sequentially to the previous ensemble's residuals, and XGBoost, LightGBM and CatBoost are the implementations that dominate real tabular work. Early stopping on a validation set is the essential guard. It sits at the top of the tabular sequence because on most tabular problems it is the answer. The method is a tuning order instead of hyperparameter flailing — learning rate and tree count first, depth next, the rest rarely.
Work through these
Boosting as sequential residual fitting
Instead of averaging independent models, each new model is fitted to what the previous ones got wrong. That sequential correction is what makes this family so strong on tabular data.
Key hyperparameters and a tuning order
A handful of settings dominate the result, and there is a sensible order to tune them in. Tuning everything at once is slower and usually worse.
Early stopping and validation sets
Stopping when a held-out score stops improving is the main defence against a boosted model overfitting. It also removes the need to guess the number of rounds.
Categorical handling differences across the three
The three widely used implementations differ most in how they handle categorical features, and that difference frequently decides which to reach for. Knowing the differences saves fighting one into behaving like another.
Sign in to keep your progress.
Free resources
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.