[Paper Review] A Large Dimensional Study of Regularized Discriminant Analysis Classifiers

Khalil Elkhalil, Abla Kammoun+3 more|arXiv (Cornell University)|Nov 1, 2017

Bayesian Methods and Mixture Models4 citations

TL;DR

This paper presents a large-dimensional asymptotic analysis of regularized linear and quadratic discriminant analysis (R-LDA and R-QDA) using random matrix theory, showing that the misclassification probability converges to a deterministic limit depending on class statistics and the ratio $ p/n $. The key contribution is a closed-form expression for this limit, enabling optimal regularization parameter tuning to minimize error in high-dimensional settings with finite samples.

ABSTRACT

This article carries out a large dimensional analysis of standard regularized discriminant analysis classifiers designed on the assumption that data arise from a Gaussian mixture model with different means and covariances. The analysis relies on fundamental results from random matrix theory (RMT) when both the number of features and the cardinality of the training data within each class grow large at the same pace. Under mild assumptions, we show that the asymptotic classification error approaches a deterministic quantity that depends only on the means and covariances associated with each class as well as the problem dimensions. Such a result permits a better understanding of the performance of regularized discriminant analsysis, in practical large but finite dimensions, and can be used to determine and pre-estimate the optimal regularization parameter that minimizes the misclassification error probability. Despite being theoretically valid only for Gaussian data, our findings are shown to yield a high accuracy in predicting the performances achieved with real data sets drawn from the popular USPS data base, thereby making an interesting connection between theory and practice.

Motivation & Objective

• To analyze the asymptotic performance of regularized LDA and QDA in high-dimensional settings where both dimension $ p $ and sample size $ n $ grow large with fixed ratio $ p/n $.
• To derive deterministic equivalents for the misclassification probability under general Gaussian mixture models with distinct means and covariances.
• To identify the growth rate conditions on class means and covariance differences under which non-trivial classification performance emerges.
• To enable optimal tuning of the regularization parameter $ \gamma $ by deriving a consistent estimator of the misclassification rate based on theoretical asymptotics.
• To validate the theoretical findings through synthetic and real-world experiments on the USPS dataset, demonstrating accurate prediction of classification error.

Proposed method

• The analysis employs tools from random matrix theory (RMT) to study the asymptotic behavior of R-LDA and R-QDA in the double asymptotic regime $ p, n \to \infty $ with $ p/n \to c \in (0, \infty) $.
• The authors derive closed-form expressions for the limiting misclassification probability under mild assumptions on the growth rates of class means and covariance differences.
• For R-LDA, the limiting error depends only on the difference in class means and the regularization parameter $ \gamma $, with a required mean difference of order $ O(1) $.
• For R-QDA, the limiting error depends on both the mean difference (requiring $ O(\sqrt{p}) $) and the spectral norm of the covariance difference, reflecting its dual use of mean and covariance information.
• A two-stage optimization method is proposed: first estimate the optimal $ \gamma $ using a Gaussian-based G-estimator, then refine it via cross-validation or testing on real data.
• The G-estimator is validated on synthetic and USPS real data, showing strong alignment with empirical testing errors across varying $ n_0 $ and $ p $.

Experimental results

Research questions

• RQ1 What is the asymptotic behavior of the misclassification probability for R-LDA and R-QDA when both dimension $ p $ and sample size $ n $ grow large with fixed ratio $ p/n $?
• RQ2 Under what growth rate conditions on class means and covariance matrices do R-LDA and R-QDA achieve non-trivial, non-degenerate classification performance?
• RQ3 How does the regularization parameter $ \gamma $ affect the asymptotic misclassification rate, and can it be optimally tuned using theoretical approximations?
• RQ4 Can the proposed theoretical estimator of the misclassification rate accurately predict the actual testing error on real-world high-dimensional data?
• RQ5 What fundamental differences exist in how R-LDA and R-QDA utilize class mean and covariance information in high-dimensional asymptotic regimes?

Key findings

• The misclassification probability for both R-LDA and R-QDA converges almost surely to a deterministic limit that depends only on class statistics and the ratio $ p/n $, derived via random matrix theory.
• R-LDA achieves perfect classification when the difference in class means is of order $ O(1) $, while R-QDA requires a mean difference of order $ O(\sqrt{p}) $ to yield non-trivial performance.
• The R-QDA classifier leverages both mean and covariance differences, but its performance is sensitive to the spectral norm of the covariance difference matrix, which must grow with $ \sqrt{p} $.
• The proposed G-estimator for the misclassification rate closely matches empirical testing errors on the USPS dataset, with prediction accuracy validated across multiple settings of $ n_0 $ and $ p $.
• For the USPS dataset, the two-stage optimization method successfully identified optimal $ \gamma $, with R-QDA achieving a minimum testing error of 0.009 when $ n_0 = 400 $ and $ p = 100 $.
• The results reveal a fundamental distinction: R-LDA primarily exploits mean differences, while R-QDA requires stronger signal in both means and covariances to outperform R-LDA in high-dimensional regimes.