What it is
XGBoost is a highly tuned implementation of gradient boosted decision trees. It builds an additive ensemble where each new tree fits the errors of the current model.
Gradient boosting core
- The model is a sum of trees, each predicting a small correction.
- Each new tree is fit to the gradient of the loss, the direction that most reduces error.
- A learning rate shrinks each tree contribution to slow learning and improve generalization.
What XGBoost adds
- It uses a second order Taylor expansion, using both gradient and curvature for sharper splits.
- A regularized objective penalizes tree complexity, the number of leaves and the size of leaf weights.
- Each split is chosen by a gain formula balancing error reduction against the complexity penalty.
Why it is fast
- It precomputes sorted feature blocks and uses histogram based split finding.
- It handles missing values with a learned default direction at each split.
- Built in shrinkage and column subsampling act like regularization.
Key idea
XGBoost grows trees that fit the loss gradient using second order curvature and an explicit complexity penalty, with shrinkage for strong generalization.