[Paper Review] Which principal components are most sensitive to distributional changes?

Martin Tveten|arXiv (Cornell University)|May 15, 2019

Advanced Statistical Process MonitoringDecision Sciences12 references3 citations

TL;DR

This paper demonstrates that in bivariate and high-dimensional data, the minor principal component (least varying projection) is most sensitive to distributional changes, especially sparse changes (e.g., one mean or variance shift). Using Hellinger distance to measure sensitivity, the authors prove theoretically and confirm via simulations that minor components outperform major components in detecting such changes, offering a principled basis for anomaly detection in high-dimensional process control.

ABSTRACT

PCA is often used in anomaly detection and statistical process control tasks. For bivariate data, we prove that the minor projection (the least varying projection) of the PCA-rotated data is the most sensitive to distributional changes, where sensitivity is defined by the Hellinger distance between distributions before and after a change. In particular, this is almost always the case if only one parameter of the bivariate normal distribution changes, i.e., the change is sparse. Simulations indicate that the minor projections are the most sensitive for a large range of changes and pre-change settings in higher dimensions as well. This motivates using the minor projections for detecting sparse distributional changes in high-dimensional data.

Motivation & Objective

• To determine which principal components are most sensitive to distributional changes in multivariate data.
• To address the limitation of prior work that focused on changes in projections rather than original data parameters.
• To investigate sensitivity under sparse changes—where only a few parameters shift—since these are common in real-world anomaly detection.
• To extend theoretical insights from bivariate to higher-dimensional settings using Monte Carlo simulations.
• To provide a foundation for designing more efficient change detection and anomaly detection methods based on component sensitivity.

Proposed method

• Theoretical analysis of sensitivity in bivariate data using the Hellinger distance between pre- and post-change marginal distributions of principal components.
• Tracing changes in original data parameters (mean, variance, correlation) through PCA to their effects on projection distributions.
• Simulating high-dimensional data with controlled changes in mean, variance, and correlation under varying sparsity levels.
• Using Monte Carlo estimation to compute expected sensitivity (Hellinger distance) across components for different change types and sparsity.
• Generating random correlation matrices uniformly using the clusterGeneration::rcorrmatrix method in R to ensure realistic pre-change structures.
• Conditioning sensitivity estimates on change type and sparsity to analyze trends across diverse scenarios.

Experimental results

Research questions

• RQ1 Which principal component is most sensitive to distributional changes when only one parameter of a bivariate normal distribution changes?
• RQ2 How does the sensitivity of principal components to distributional changes depend on the pre-change correlation structure?
• RQ3 Does the minor component remain the most sensitive under sparse changes in higher-dimensional data?
• RQ4 How does the sensitivity of principal components vary when changes affect means, variances, or correlations in high-dimensional settings?
• RQ5 Can the minor component be reliably used as a detection target for anomaly or change detection in high-dimensional data?

Key findings

• In bivariate data, the minor projection is the most sensitive to distributional changes when only one mean, one variance, or the correlation changes.
• The principal component is more sensitive than the minor component only when a variance decreases and the correlation is not close to 1.
• When both means change, sensitivity depends on the relative direction and magnitude of the change, with both components equally sensitive if changes are symmetric.
• For sparse changes, the minor component is more sensitive on average, making it a strong candidate for anomaly detection in high-dimensional data.
• Simulations confirm that the minor components remain the most sensitive on average in high-dimensional settings (D=20 and D=100), across changes in mean, variance, and correlation.
• Despite average trends, sensitivity varies significantly depending on the specific pre-change correlation matrix and the exact change vector, indicating context dependence.