Chapter 09: Tuning
This chapter introduces and formalizes the problem of hyperparameter tuning. We cover basic techniques such as grid search and random search as well as more advanced techniques like evolutionary algorithms, model-based optimization and multi-fidelity optimization.
-
Chapter 09.00: Tuning & Nested Resampling: In a Nutshell
In this nutshell chunk, we explore tuning and nested resampling, focusing on their roles in evaluating and optimizing the performance of machine learning models.
-
Chapter 09.01: Introduction
While model parameters are optimized during training, hyperparameters must be specified in advance. In this section, we will motivate why it is crucial to find good values for, i.e. to tune, these hyperparameters.
-
Chapter 09.02: Problem Definition
Hyperparameter tuning is the process of finding good model hyperparameters. In this section we formalize the problem of tuning and explain why tuning is computationally hard.
-
Chapter 09.03: Basic Techniques
In this section we familiarize ourselves with two simple but popular tuning strategies, namely grid search and random search, and discuss their advantages and disadvantages.
-
Chapter 09.04: Advanced Tuning Techniques
Besides grid search and random search there are several more advanced techniques for hyperparameter optimization. In this section we focus on model based optimization methods such as Bayesian optimization. Furthermore, we look into multi-fidelity methods such as the hyperband algorithm.
-
Chapter 09.05: Pipelines and AutoML
Some aspects of the machine learning lifecycle can be automated via AutoML. In this section we look into pipelines as part of AutoML and how (HPO-) pipelines can be represented as directed acyclic graphs (DAGs).
- Tuning: Further Material