[Paper Review] Linear and Quadratic Discriminant Analysis: Tutorial

Benyamin Ghojogh, Mark Crowley|arXiv (Cornell University)|Jun 1, 2019

Face and Expression Recognition27 references83 citations

TL;DR

This tutorial derives LDA and QDA for binary and multi-class settings, links them to metric learning and related methods, and discusses parameter estimation and equivalence to Fisher FDA. It also provides simulations for illustration.

ABSTRACT

This tutorial explains Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA) as two fundamental classification methods in statistical and probabilistic learning. We start with the optimization of decision boundary on which the posteriors are equal. Then, LDA and QDA are derived for binary and multiple classes. The estimation of parameters in LDA and QDA are also covered. Then, we explain how LDA and QDA are related to metric learning, kernel principal component analysis, Mahalanobis distance, logistic regression, Bayes optimal classifier, Gaussian naive Bayes, and likelihood ratio test. We also prove that LDA and Fisher discriminant analysis are equivalent. We finally clarify some of the theoretical concepts with simulations we provide.

Motivation & Objective

• Explain the optimization and boundary formation for class separation in LDA and QDA.
• Derive LDA and QDA for binary and multi-class classification under Gaussian assumptions.
• Show estimation procedures for priors, means, and covariances in LDA/QDA.
• Relate LDA/QDA to metric learning, Mahalanobis distance, kernel methods, FDA, and Bayes optimal classifier.
• Provide simulations to illustrate theoretical concepts and boundary behavior.

Proposed method

• Formulate the decision boundary by equating posteriors and deriving Gaussian-based discriminant equations.
• Derive the linear decision boundary for LDA under equal covariance and Gaussian assumptions.
• Derive the quadratic decision boundary for QDA when class covariances differ.
• Show parameter estimation for priors, class means, and class-specific covariances (MLE/MOM and unbiased variants).
• Connect LDA/QDA to metric learning via Mahalanobis distance and SVD-based data transformations.
• Discuss equivalence and relation to FDA, kernel PCA, and Bayes/LRT concepts.

Experimental results

Research questions

• RQ1 What is the form of the decision boundary under Gaussian assumptions for LDA and QDA?
• RQ2 How do priors, means, and covariance estimates affect the LDA/QDA boundaries and classifications?
• RQ3 What are the connections between LDA/QDA and related methods like metric learning, Mahalanobis distance, FDA, and kernel methods?
• RQ4 Under what conditions do LDA and QDA reduce to simple Euclidean or metric-based classifications?
• RQ5 How can LDA/QDA be interpreted as transformations that induce metric learning or manifold learning.

Key findings

• LDA yields a linear decision boundary when class covariances are equal, derived by equating posterior probabilities using Gaussian likelihoods.
• QDA yields a quadratic decision boundary when class covariances are unequal, with the boundary defined by a quadratic form in x.
• Both LDA and QDA can be viewed as learning a metric (Mahalanobis-type distance) via class-specific covariances, or via a data transformation that normalizes covariances to identity.
• With equal covariances and equal priors, both methods reduce to simple distance-based classification (Euclidean distance to class means).
• The priors scale the distance to class means, shifting the decision boundary according to class proportions.
• Parameter estimation for LDA/QDA follows standard ML/MOM approaches for means and covariances, with priors estimated by class frequencies.