[Paper Review] The curse of isotropy: from principal components to principal subspaces

Tom Szwagier, Xavier Pennec|arXiv (Cornell University)|Jul 28, 2023

Spectroscopy and Chemometric AnalysesChemistry3 citations

TL;DR

This paper introduces Stratified Principal Component Analysis (SPCA), a generalization of probabilistic PCA that models covariance matrices with repeated eigenvalues using flag manifolds, enabling a principled tradeoff between model complexity and goodness-of-fit. It shows that adjacent eigenvalues with gaps below 21% require over 1000 samples to distinguish, making block-averaging a better choice in small-data regimes.

ABSTRACT

Principal component analysis is a ubiquitous tool in exploratory data analysis. It is widely used by applied scientists for visualization and interpretability purposes. We raise an important issue (the curse of isotropy) about the interpretation of principal components with close eigenvalues. They may indeed suffer from an important rotational variability, which is a pitfall for interpretation. Through the lens of a probabilistic covariance model parameterized with flags of subspaces, we show that the curse of isotropy cannot be overlooked in practice. In this context, we propose to transition from ill-defined principal components to more-interpretable principal subspaces. The final methodology (principal subspace analysis) is extremely simple and shows promising results on a variety of datasets from different fields.

Motivation & Objective

• To address the practical identifiability of eigenvectors in PCA when eigenvalues are close, especially in small-sample regimes.
• To extend probabilistic PCA (PPCA) by allowing repeated eigenvalues in the signal space, not just the noise space, to improve model parsimony.
• To provide a geometric framework using flag manifolds that unifies the parameterization of covariance models with eigenvalue multiplicities.
• To develop efficient, consistent model selection heuristics for SPCA by exploiting the stratified hierarchy of eigenvalue multiplicities.
• To empirically validate that grouping adjacent eigenvalues with small gaps improves the complexity/goodness-of-fit tradeoff compared to standard PPCA.

Proposed method

• Proposes a family of covariance models—SPCA—where eigenvalues are grouped into blocks of equal values, parameterized by flag manifolds.
• Derives an explicit maximum likelihood estimate via eigenvalue decomposition of the sample covariance matrix followed by block-averaging of adjacent eigenvalues.
• Uses the stratification of eigenvalue multiplicities (via compositions of integers) to define a partial order on SPCA models, enabling hierarchical model selection.
• Applies the Bayesian Information Criterion (BIC) to compare models, with a penalty term that accounts for the number of free parameters derived from the flag manifold geometry.
• Develops a hierarchical clustering heuristic that first merges eigenvalues within the same multiplicity block, then merges blocks by increasing gap size.
• Proves asymptotic consistency of both hierarchical clustering and maximum likelihood selection, showing that the true model is recovered as sample size increases.

Figure 1 : SPCA generative model ( 6 ), assuming that the observed data was first sampled from a sequence of independent lower dimensional normal latent variables, then linearly mapped to mutually orthogonal subspaces and finally shifted and added an isotropic Gaussian noise. The resulting density i

Experimental results

Research questions

• RQ1 Under what conditions can two adjacent sample eigenvalues be statistically distinguished, and how many samples are required?
• RQ2 Does grouping adjacent eigenvalues with small gaps lead to a better complexity/goodness-of-fit tradeoff than standard PPCA in small-sample settings?
• RQ3 Can a unified geometric framework based on flag manifolds provide a principled parameterization of covariance models with repeated eigenvalues?
• RQ4 Is there an efficient, consistent model selection procedure for the exponentially growing family of SPCA models?
• RQ5 How does the number of free parameters in SPCA models depend on the eigenvalue multiplicity structure?

Key findings

• A pair of adjacent eigenvalues with a relative gap below 21% requires at least 1000 samples to be reliably distinguished using BIC.
• When the gap between adjacent eigenvalues is below the 21% threshold, a model with equal eigenvalues and a two-dimensional eigenspace is more optimal under BIC.
• SPCA models achieve a better complexity/goodness-of-fit tradeoff than standard PPCA on both synthetic and real datasets, especially in low-sample regimes.
• The hierarchical clustering heuristic for model selection is asymptotically consistent, correctly recovering the true model type as sample size increases.
• The maximum likelihood estimator for SPCA is explicitly computable via eigenvalue decomposition and block-averaging, with the number of free parameters determined by the flag manifold structure.
• The geometric interpretation via flag manifolds provides a unifying framework that generalizes both PPCA and isotropic PPCA (IPPCA), with a clear parameter count derived from the stratification.

Figure 2 : Plot of the inverse threshold function $\delta^{-1}$ of Proposition 3 , corresponding to the minimal number of samples needed to distinguish two adjacent eigenvalues separated by a given relative eigengap.