Chapter 13: Information Theory

We introduce entropy, which expresses the expected information for discrete random variables, as a central concept in information theory.

We continue our discussion about entropy and introduce joint entropy, the uniqueness theorem and the maximum entropy principle.

In this section, we extend the definition of entropy to the continuous case.

The Kullback-Leibler divergence (KL) is an important quantity for measuring the difference between two probability distributions. We discuss different intuitions for KL and relate it to risk minimization and likelihood ratios.

We introduce cross-entropy as a further information-theoretic concept and discuss the connection between entropy, cross-entropy, and Kullback-Leibler divergence.

In this section, we discuss how information-theoretic concepts are used in machine learning and demonstrate the equivalence of KL minimization and maximum likelihood maximization, as well as how (cross-)entropy can be used as a loss function.

Information theory also provides means of quantifying relations between two random variables that extend the concept of (linear) correlation. We discuss joint entropy, conditional entropy, and mutual information in this context.

Information theory also provides means of quantifying relations between two random variables that extend the concept of (linear) correlation. We discuss joint entropy, conditional entropy, and mutual information in this context.

In this section, we introduce source coding and discuss how entropy can be understood as optimal code length.

In this section, we continue our discussion on source coding and its relation to entropy.

In this deep dive, we discuss the invariance of MI under certain reparametrizations.