Chapter 18: Boosting
This chapter introduces boosting as a sequential ensemble method that creates powerful committees from different kinds of base learners.
-
Chapter 18.01: Introduction to Boosting / AdaBoost
In this section, we introduce the pioneering AdaBoost algorithm.
-
Chapter 18.02: Boosting Concept
In this section, we discuss the general boosting principle: performing gradient descent in function space by repeatedly fitting new base learner components to the current pseudo-residuals.
-
Chapter 18.03: Boosting Illustration
We show several illustrative regression examples to visualize the boosting principle.
-
Chapter 18.04: Boosting Regularization
Powerful boosting learners tend to overfit. We discuss the number of iterations, base learner complexity, and shrinkage as countermeasures.
-
Chapter 18.05: Boosting for Classification
We introduce boosting algorithms for both binary and multiclass classification with several examples.
-
Chapter 18.06: Gradient Boosting with Trees I
We discuss trees as the most popular base learners in gradient boosting, with special emphasis on model structure and interaction depth.
-
Chapter 18.07: Gradient Boosting with Trees II
We explain how terminal coefficients are found in a risk-minimal manner and briefly discuss tree-based boosting for multiclass problems.
-
Chapter 18.08: XGBoost
We introduce XGBoost, a highly efficient, tree-based boosting system with additional regularizers.
-
Chapter 18.09: Component Wise Boosting Basics 1
We introduce the concept of CWB, common base learners and built-in feature selection.
-
Chapter 18.10: Component Wise Boosting Basics 2
We explain the handling of categorical features and of the intercept and introduce a practical example.
-
Chapter 18.11: CWB and GLMs
We explain the relationship between CWB and GLMs.
-
Chapter 18.12: Advanced CWB
We explain the details of nonlinear BLs and splines, decomposition for splines, fair base learner selection and feature importance and PDPs.