Topic 03: Regularization
This chapter introduces and discusses regularization techniques for neural networks, which help prevent overfitting and improve generalization. It provides an introduction and geometric intuition of L2-regularization. Additionally, it introduces dropout, a method of randomly deactivating neurons during training to enhance robustness, and early stopping, which monitors validation performance to halt training when overfitting begins.
-
Chapter 03.01: Basic Regularization
In this section we discuss regularized cost functions, norm penalties, weight decay, and equivalence with constrained optimization.
-
Chapter 03.02: Ridge Regularization
In this section, we introduce Ridge regression as a key approach to regularizing linear models (Material provided by: I2ML/SL lecture. )
-
Chapter 03.03: Regularization in Non-Linear Models
In this section, we introduce regularization in non-linear models like neural networks. (Material provided by: I2ML/SL lecture. )
-
Chapter 03.04: Geometric Analysis of L2 Regularization and Weight Decay
In this section, we provide a geometric understanding of \(L2\) regularization, showing how parameters are shrunk according to the eigenvalues of the Hessian of empirical risk, and discuss its correspondence to weight decay. (Material provided by: I2ML/SL lecture. )
-
Chapter 03.05: Early Stopping
In this section, we introduce early stopping and show how it can act as a regularizer.
-
Chapter 03.06: Regularization
In this section, we explain ensemble methods, dropout and data augmentation.