Chapter 03: Supervised Classification

In this nutshell chunk, we delve into the basics of supervised classification, where we train machines to categorize input data into predefined labels.

In classification, the task is to predict a categorical (binary or multiclass) label. In this section, we illustrate the concept of classification with some typical examples.

Although we are primarily interested in actual class labels, classification models usually output scores or probabilities first. We will explain why, introduce the concepts of decision regions and decision boundaries, and discern two fundamental approaches to constructing classifiers: the generative approach and the discriminant approach.

Linear classifiers are an essential subclass of classification models. This section provides the definition of a linear classifier and depicts differences between linear and non-linear decision boundaries.

Logistic regression is a discriminant approach toward constructing a classifier. We will motivate logistic regression via the logistic function, define the log-loss for optimization and illustrate the approach in 1D and 2D.

Discriminant analysis is a generative approach toward constructing a classifier. We distinguish between linear (LDA) and quadratic (QDA) discriminant analysis, where the latter is a more flexible approach subsuming the first.

Naive Bayes is a generative approach based on an assumption of conditional independence across features and closely related to discriminant analysis.