Chapter 09: Tuning

In this nutshell chunk, we explore tuning and nested resampling, focusing on their roles in evaluating and optimizing the performance of machine learning models.

While model parameters are optimized during training, hyperparameters must be specified in advance. In this section, we will motivate why it is crucial to find good values for, i.e. to tune, these hyperparameters.

Hyperparameter tuning is the process of finding good model hyperparameters. In this section we formalize the problem of tuning and explain why tuning is computationally hard.

In this section we familiarize ourselves with two simple but popular tuning strategies, namely grid search and random search, and discuss their advantages and disadvantages.

Besides grid search and random search there are several more advanced techniques for hyperparameter optimization. In this section we focus on model based optimization methods such as Bayesian optimization. Furthermore, we look into multi-fidelity methods such as the hyperband algorithm.

Some aspects of the machine learning lifecycle can be automated via AutoML. In this section we look into pipelines as part of AutoML and how (HPO-) pipelines can be represented as directed acyclic graphs (DAGs).

Further Material Bischl, Bernd, et al. “Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges.” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery (2021): e1484. This paper goes beyond grid search and random search and reviews important automatic hyperparameter optimization (HPO) methods, provides practical recommendations for conducting HPO, and discusses HPO algorithms, performance evaluation, combination with machine learning pipelines, runtime improvements, and parallelization. mlr3 Practical Tuning Series: These notebooks are a step-by-step hands-on tutorial on how to tune ML models with mlr3.