Topic 03: Regularization
This chapter introduces the concept of regularization and discusses common regularization techniques in more depth.
-
Chapter 03.01: Basic Regularization
In this section we discuss regularized cost functions, norm penalties, weight decay, and equivalence with constrained optimization.
-
Chapter 03.02: Regularization in Non-Linear Models and Bayesian Priors
In this section, we motivate regularization from a Bayesian perspective.
-
Chapter 03.03: Geometric Analysis of L2 Regularization and Weight Decay
In this section, we provide a geometric understanding of \(L2\) regularization, showing how parameters are shrunk according to the eigenvalues of the Hessian of empirical risk, and discuss its correspondence to weight decay.
-
Chapter 03.04: Early Stopping
In this section, we introduce early stopping and show how it can act as a regularizer.
-
Chapter 03.05: Regularization
In this section, we explain ensemble method, dropout and data augmentation.