Chapter 10: Nested Resampling
This chapter first defines the untouched-test-set principle and proceeds to explain the concepts of train-validation-test split and nested resampling.
-
Chapter 10.01: Motivation
Selecting the best model from a set of candidates is an important part of most machine learning problems. By examining an instructive and problematic example, we introduce the untouched-test-set principle.
-
Chapter 10.02: Training - Validation - Testing
The simplest method to achieve an untouched test set is a 3-way split: the models are first trained on the training set and then evaluated and compared on the validation set. After selecting the best model, the final performance will be evaluated on the test set.
-
Chapter 10.03: Nested Resampling
In this section, we will explain why and how nested resampling is done.