[Paper Review] High-dimensional principal component analysis with heterogeneous missingness

Ziwei Zhu, Tengyao Wang+1 more|arXiv (Cornell University)|Jun 28, 2019

Sparse and Compressive Sensing Techniques39 references22 citations

TL;DR

This paper proposes primePCA, a novel iterative method for high-dimensional principal component analysis under heterogeneous missingness, where missingness probabilities vary across features. Starting from an inverse-probability weighted estimator, primePCA alternately imputes missing entries via projection onto the current estimate's column space and updates the estimate via SVD of the imputed matrix. The method achieves geometric convergence to the true principal components in the noiseless case under a novel incoherence condition, with theoretical guarantees depending on average, not worst-case, missingness properties.

ABSTRACT

We study the problem of high-dimensional Principal Component Analysis (PCA) with missing observations. In a simple, homogeneous observation model, we show that an existing observed-proportion weighted (OPW) estimator of the leading principal components can (nearly) attain the minimax optimal rate of convergence, which exhibits an interesting phase transition. However, deeper investigation reveals that, particularly in more realistic settings where the observation probabilities are heterogeneous, the empirical performance of the OPW estimator can be unsatisfactory; moreover, in the noiseless case, it fails to provide exact recovery of the principal components. Our main contribution, then, is to introduce a new method, which we call primePCA, that is designed to cope with situations where observations may be missing in a heterogeneous manner. Starting from the OPW estimator, primePCA iteratively projects the observed entries of the data matrix onto the column space of our current estimate to impute the missing entries, and then updates our estimate by computing the leading right singular space of the imputed data matrix. We prove that the error of primePCA converges to zero at a geometric rate in the noiseless case, and when the signal strength is not too small. An important feature of our theoretical guarantees is that they depend on average, as opposed to worst-case, properties of the missingness mechanism. Our numerical studies on both simulated and real data reveal that primePCA exhibits very encouraging performance across a wide range of scenarios, including settings where the data are not Missing Completely At Random.

Motivation & Objective

• To address the limitations of existing inverse-probability weighted (IPW) estimators in high-dimensional PCA when missingness is heterogeneous.
• To develop a method robust to realistic missingness patterns where some features are observed more frequently than others.
• To establish theoretical guarantees for principal component recovery that depend on average, rather than worst-case, missingness mechanisms.
• To provide a practical and theoretically grounded algorithm for PCA in the presence of missing data in high-dimensional settings.

Proposed method

• Proposes primePCA, an iterative algorithm that alternates between imputing missing entries and updating the principal component estimate.
• Imputes missing entries by projecting the observed entries of the data matrix onto the column space of the current estimate.
• Updates the estimate by computing the leading right singular space of the imputed data matrix at each iteration.
• Introduces a novel incoherence condition on the principal components to ensure feasibility of recovery under heterogeneous missingness.
• Employs inverse-probability weighting as an initial estimator to initialize the iterative procedure.
• Theoretical analysis shows geometric convergence of the error to zero in the noiseless case when signal strength is sufficient.

Experimental results

Research questions

• RQ1 Can existing IPW estimators for high-dimensional PCA achieve exact recovery in the noiseless case under heterogeneous missingness?
• RQ2 What structural conditions on the principal components and missingness mechanism enable consistent recovery in high-dimensional PCA with missing data?
• RQ3 How does the interaction between heterogeneous missingness and low-rank structure affect the feasibility of PCA recovery?
• RQ4 Can an iterative refinement procedure improve upon the initial IPW estimator in terms of convergence rate and accuracy?
• RQ5 What are the finite-sample and asymptotic properties of the proposed method under realistic missingness patterns?

Key findings

• primePCA achieves geometric convergence to the true principal components in the noiseless case when the signal strength is not too small and the incoherence condition holds.
• Theoretical guarantees depend on average, not worst-case, properties of the missingness mechanism, making the method robust to heterogeneous patterns.
• In the noiseless case, the IPW estimator fails to provide exact recovery, highlighting the need for iterative refinement.
• The method significantly outperforms the IPW estimator in numerical studies across simulated and real-world datasets, including the Million Song Dataset.
• Theoretical bounds show that primePCA achieves minimax optimal rates up to logarithmic factors under appropriate conditions.
• The incoherence condition is essential for ensuring that the principal components are not aligned with the missingness pattern, enabling consistent recovery.