Chapter 03.03: Geometric Analysis of L2 Regularization and Weight Decay

In this section, we provide a geometric understanding of \(L2\) regularization, showing how parameters are shrunk according to the eigenvalues of the Hessian of empirical risk, and discuss its correspondence to weight decay.

Lecture video

Lecture slides

Download »slides-regu-geom-l2-wdecay.pdf«

• « Chapter 03.02: Regularization in Non-Linear Models and Bayesian Priors
• Chapter 03.04: Early Stopping »