Topic 1: Introduction

In this section, we introduce the relationship of DL and ML, give a basic intro about feature learning, and discuss the use-cases and data types for DL methods.

In this section we explain the graphical representation of a single neuron and describe affine transformations and non-linear activation functions. Moreover, we talk about the hypothesis space of a single neuron and name some typical loss functions.

In this subsection, we present the XOR problem, a famous example a single neuron can not solve but a single hidden layer net can solve.

In this subchapter, we introduce the architecture of single hidden layer neural networks and discuss the advantages of hidden layers. In addition, we present some typical (non-linear) activation functions.

In this subsection, we discuss neural network architectures for multi-class classification, the softmax activation function as well as the softmax loss.

In this subchapter, we present the architecture of deep neural networks and introduce deep neural networks as chained functions.

In this section we explain the compact representation of neural networks, the vector notation for neuron layers and both the vector and matrix notation for bias and weight parameters.

Here we present the universal approximation theorem for one-hidden-layer neural networks. In addition, we discuss the pros and cons of a low approximation error.

In this subsection we present an overview of the history of DL development.