[Paper Review] High-Dimensional Regularized Discriminant Analysis

John A. Ramey, Caleb K. Stein+2 more|arXiv (Cornell University)|Feb 3, 2016

Gene expression and cancer classification40 references3 citations

TL;DR

This paper proposes High-Dimensional Regularized Discriminant Analysis (HDRDA), a computationally efficient and interpretable classifier for small-sample, high-dimensional data where p > N. By reparameterizing RDA to pool training observations into class covariance estimates and reducing dimensionality via null-space elimination, HDRDA achieves linear computational complexity in p, outperforming RDA and other classifiers in accuracy and speed—being up to 502.786× faster on p=5000 features.

ABSTRACT

Regularized discriminant analysis (RDA), proposed by Friedman (1989), is a widely popular classifier that lacks interpretability and is impractical for high-dimensional data sets. Here, we present an interpretable and computationally efficient classifier called high-dimensional RDA (HDRDA), designed for the small-sample, high-dimensional setting. For HDRDA, we show that each training observation, regardless of class, contributes to the class covariance matrix, resulting in an interpretable estimator that borrows from the pooled sample covariance matrix. Moreover, we show that HDRDA is equivalent to a classifier in a reduced-feature space with dimension approximately equal to the training sample size. As a result, the matrix operations employed by HDRDA are computationally linear in the number of features, making the classifier well-suited for high-dimensional classification in practice. We demonstrate that HDRDA is often superior to several sparse and regularized classifiers in terms of classification accuracy with three artificial and six real high-dimensional data sets. Also, timing comparisons between our HDRDA implementation in the sparsediscrim R package and the standard RDA formulation in the klaR R package demonstrate that as the number of features increases, the computational runtime of HDRDA is drastically smaller than that of RDA.

Motivation & Objective

• Address the computational infeasibility and lack of interpretability of traditional Regularized Discriminant Analysis (RDA) in high-dimensional, small-sample settings (p > N).
• Develop a classifier that maintains RDA’s regularization benefits while enabling efficient computation and clear interpretation of how training observations contribute to covariance estimation.
• Reduce the dimensionality of matrix operations in RDA by eliminating redundant null-space components, enabling linear-time computation in the number of features.
• Demonstrate that HDRDA achieves superior classification accuracy compared to RDA, sparse classifiers, and random forests on real and simulated high-dimensional datasets.
• Provide an efficient implementation in the sparsediscrim R package, showing drastic speed improvements over standard RDA in model selection.

Proposed method

• Reparameterize RDA using a biased covariance estimator that pools individual class covariance matrices with the pooled sample covariance matrix, enabling interpretability of each training observation’s contribution.
• Shrink the resulting pooled estimator toward a scaled identity matrix to ensure positive definiteness and stability.
• Apply dimension reduction by discarding matrix operations in the null space of the pooled covariance matrix, which are redundant for classification and reduce dimension to approximately the sample size q.
• Use singular value decomposition (SVD) to identify and eliminate the p−N zero eigenvalues in the pooled covariance matrix, reducing the problem to a lower-dimensional space.
• Formulate the HDRDA decision rule in the reduced space, where matrix inverses and determinants are computed on small, nonsingular matrices, enabling linear-time computation in p.
• Implement an efficient grid search with cross-validation for tuning parameters using the sparsediscrim R package, leveraging the reduced dimensionality for fast model selection.

Experimental results

Research questions

• RQ1 Can a regularized discriminant analysis classifier be made both interpretable and computationally efficient in high-dimensional, small-sample settings (p > N)?
• RQ2 Does replacing the standard RDA formulation with a dimension-reduced, null-space-eliminated approach preserve classification accuracy while drastically reducing computational cost?
• RQ3 How does HDRDA’s performance compare to RDA, random forests, and other sparse or diagonal classifiers on real and simulated high-dimensional datasets?
• RQ4 To what extent do existing shrinkage estimators (e.g., Srivastava & Kubokawa, Rao & Mitra) remain effective under the HDRDA framework when p > N?
• RQ5 Can the HDRDA model selection process be accelerated sufficiently to make it practical for large-scale high-dimensional data, such as gene expression or imaging data?

Key findings

• HDRDA achieves a 502.786× speedup over standard RDA in model selection when p = 5000, reducing average runtime from 24.933 minutes to 2.979 seconds.
• HDRDA consistently outperforms RDA, random forests, and several sparse classifiers (e.g., Pang, Tong, Witten) in classification accuracy across six real high-dimensional datasets and three simulated datasets.
• The HDRDA decision rule is invariant to adjustments of the approximately p−N zero eigenvalues in the pooled covariance matrix, confirming that null-space operations are redundant and can be safely discarded.
• HDRDA is robust to outliers and maintains high performance even when other classifiers like the penalized linear discriminant analysis (Witten & Tibshirani, 2011) fail due to numerical instability.
• The HDRDA classifier is equivalent to a classifier in a reduced feature space of dimension approximately equal to the training sample size, enabling linear-time matrix operations in p.
• The HDRDA implementation in the sparsediscrim R package is computationally viable for large-scale high-dimensional data, replacing RDA as the practical choice in such settings.