Chapters

This chapter introduces the basic concepts of Machine Learning. We focus on supervised learning, explain the difference between regression and classification, show how to evaluate and compare Machine Learning models and formalize the concept of learning.

This chapter treats the supervised regression task in more detail. We will see different loss functions for regression, how a linear regression model can be used from a Machine Learning perspective, and how to extend it with polynomials for greater flexibility.

This chapter treats the supervised classification task in more detail. We will see examples of binary and multiclass classification and the differences between discriminative and generative approaches. In particular, we will address logistic regression, discriminant analysis and naive Bayes classifiers.

This chapter treats the challenge of evaluating the performance of a model. We will introduce different performance measures for regression and classification tasks, explain the problem of overfitting as well as the difference between training and test error, and, lastly, present a variety of resampling techniques.

This chapter addresses \(k\)-nearest neighbors, a distance-based algorithm suited to both regression and classification. Predictions are made based upon neighboring observations, assuming feature similarity translates to target similarity.

This chapter introduces Classification and Regression Trees (CART), a well-established machine learning procedure. We explain the main idea and give details on splitting criteria, discuss computational aspects of growing a tree, and illustrate the idea of stopping criteria and pruning.

This chapter introduces bagging as a method to increase the performance of trees (or other base learners). A modification of bagging leads to random forests. We explain the main idea of random forests, benchmark their performance with the methods seen so far and show how to quantify the impact of a single feature on the performance of the random forest as well as how to compute proximities between observations.

This chapter introduces the basic concepts of neural networks. We integrated chapters from our course on Deep Learning in order to be able to use (simple) neural networks for supervised ML on tabular data.

This chapter introduces and formalizes the problem of hyperparameter tuning. We cover basic techniques such as grid search and random search as well as more advanced techniques like evolutionary algorithms, model-based optimization and multi-fidelity optimization.

This chapter first defines the untouched-test-set principle and proceeds to explain the concepts of train-validation-test split and nested resampling.

This chapter revisits the theory of risk minimization, providing more in-depth analysis on established losses and the connection between empirical risk minimization and maximum likelihood estimation. We also introduce some more advanced loss functions for regression and classification.

This chapter treats the multiclass case of classification. Tasks with more than two classes preclude the application of some techniques studied in the binary scenario and require an adaptation of loss functions.

This chapter covers basic information-theoretic concepts and discusses their relation to machine learning.

Frequently, our intuition developed in low-dimensional spaces does not generalize to higher dimensions. This chapter introduces the phenomenon of the curse of dimensionality and discusses its effects on the behavior of machine learning models.

Regularization is a vital tool in machine learning to prevent overfitting and foster generalization ability. This chapter introduces the concept of regularization and discusses common regularization techniques in more depth.

This chapter introduces the linear support vector machine (SVM), a linear classifier that finds decision boundaries by maximizing margins to the closest data points, possibly allowing for violations to a certain extent.

Many classification problems warrant nonlinear decision boundaries. This chapter introduces nonlinear support vector machines as a crucial extension to the linear variant.

This chapter introduces boosting as a sequential ensemble method that creates powerful committees from different kinds of base learners.

This chapter introduces Gaussian processes as a model class. Gaussian processes are non-parametric approaches with ubiquitous application that model entire distributions in function space.

This chapter introduces techniques for learning on imbalanced datasets.

This chapter introduces multitarget learning techniques.

This chapter introduces online learning.

This chapter introduces feature selection, i.e., dinding a well-performing, hopefully small set of features for a task.

This section introduces basic concepts and implementations using Python and in particular sklearn.

For an introduction to the R package mlr3 we recommend walking through some chapters of the mlr3 book as summarized in this document. After some basic concepts, this focuses on resampling, tuning and pipelines.