Chapter 5: Feature Importance
Feature importance (FI) methods rank features by how much they contribute to the predictive performance or prediction variance of the model. With a loss-based focus this section deals with the FI methods PFI, CFI, SAGE and LOCO.
-
Chapter 5.1: Introduction to Loss-based Feature Importance
Methods belonging to this category aim to rank the features according to their influence on the predictive performance of an ML model. Depending on the interpretation goal, these methods are more or less suitable.
-
Chapter 5.2: Permutation Feature Importance (PFI)
Developed in 2001 for Random Forests, PFI nowadays is a model-agnostic feature importance measure. Since it tends to assign higher importance to correlated features, the test method PIMP is also discussed in this section.
-
Chapter 5.3: Conditional Feature Importance (CFI)
As PFI struggles when data contains correlated features, conditional feature importance (CFI) tackles this problem by using a conditional sampling strategy. This section describes the theoretical basis of this process and gives a guideline how to interpret CFI results.
-
Chapter 5.4: Leave One Covariate Out (LOCO)
This section gives a formal definition and an interpretation guideline for the IML method LOCO (Leave One Covariate Out) which follows a different approach than PFI and CFI as model refitting is used.