Chapter 22: Preprocessing

We place feature engineering in the ML workflow and pipeline context, explain how it addresses common data science challenges like skewed distributions, high-cardinality categorical features, and missingness, and discuss why it still matters in the age of deep learning.

We cover one-hot encoding and target/impact encoding for categorical features.

We discuss feature transformations including normalization (standardization) and Box-Cox transformations for stabilizing variance.

We explain why and when to transform the target variable using log, Box-Cox, and Yeo-Johnson transformations, and introduce the transform-fit-invert pipeline.

We explain the concept of train-test leakage, why it leads to overoptimistic performance estimates, and how to avoid it in preprocessing pipelines.

We motivate the problem of missing data, show how to visualize missingness in a dataset, and outline the main options for handling it (dropping observations, dropping features, models that tolerate missing values, and imputation).

We cover simple univariate imputation strategies (mean, median, mode, out-of-range constants), the use of missingness indicator variables, the distribution shift caused by constant imputation, and imputation by sampling from the empirical feature distribution.

We introduce model-based imputation, where a surrogate model is trained on the other features to predict the missing values, and discuss its drawbacks: sensitivity to the choice of surrogate model, the need for the surrogate to handle missing values itself, and per-feature hyperparameter tuning.

We introduce time series forecasting, explain how to apply tree-based models to time series, and cover detrending and multi-step forecasting.

We cover feature engineering for time series including calendar features, cyclic encoding, lagged and rolling features, and feature importance.

We discuss time series cross-validation (expanding and sliding window), evaluation metrics, baselines, and practical guidance.