Topic 04: Optimization Part 2

In this section, we summarize several of the most prominent challenges regarding training of deep neural networks such as Ill-Conditioning Local Minima, Saddle Points, Cliffs and Exploding Gradients.

In this section we introduce several advanced techniques for optimization of neural network such as learning rate schedules, adaptive learning rates, and batch normalization.

We explain challenges in optimization related to action function and introduce activation for hidden and output units.

This part describe why initialization is important and explain weight and bias initialization.