Chapter 15: Regularization

In this section, we revisit overfitting and introduce regularization as a remedy.

We introduce Ridge regression as a key approach to regularizing linear models.

We introduce Lasso regression as a key approach to regularizing linear models.

This section provides a detailed comparison between Lasso and Ridge regression.

In this section, we introduce the elastic net as a combination of Ridge and Lasso regression and discuss regularization for logistic regression.

In this section, we introduce other regularization approaches besides the important special cases \(L1\) and \(L2\).

In this section, we demonstrate regularization in non-linear models like neural networks.

In this section, we motivate regularization from a Bayesian perspective, showing how different penalty terms correspond to different Bayesian priors.

In this section, we show that L2 regularization with gradient descent is equivalent to weight decay and see how weight decay changes the optimization trajectory.

In this section, we provide a geometric understanding of \(L2\) regularization, showing how parameters are shrunk according to the eigenvalues of the Hessian of empirical risk, and discuss its correspondence to weight decay.

In this section, we provide a geometric understanding of \(L1\) regularization and show that it encourages sparsity in the parameter vector.

In this section, we introduce early stopping and show how it can act as a regularizer.

In this section, we consider Ridge regression as row-augmentation and as minimizing risk under feature noise. We also discuss the bias-variance tradeoff.

In this section, we prove the previously stated proposition regarding soft-thresholding and L1 regularization.