[Paper Review] PCA, SVD, and Centering of Data

Donggun Kim, Kisung You|arXiv (Cornell University)|Jul 27, 2023

Spectroscopy and Chemometric Analyses4 citations

TL;DR

This paper rigorously analyzes the impact of data centering in SVD-based PCA, showing that without centering, the first right singular vector captures the mean direction, and the resulting PCA embeddings can be approximately aligned with true PCA under specific spectral conditions. The key contribution is a theoretical characterization of the discrepancy between centered and uncentered SVD-based PCA, offering bounds on the error introduced by omitting centering.

ABSTRACT

The research detailed in this paper scrutinizes Principal Component Analysis (PCA), a seminal method employed in statistics and machine learning for the purpose of reducing data dimensionality. Singular Value Decomposition (SVD) is often employed as the primary means for computing PCA, a process that indispensably includes the step of centering - the subtraction of the mean location from the data set. In our study, we delve into a detailed exploration of the influence of this critical yet often ignored or downplayed data centering step. Our research meticulously investigates the conditions under which two PCA embeddings, one derived from SVD with centering and the other without, can be viewed as aligned. As part of this exploration, we analyze the relationship between the first singular vector and the mean direction, subsequently linking this observation to the congruity between two SVDs of centered and uncentered matrices. Furthermore, we explore the potential implications arising from the absence of centering in the context of performing PCA via SVD from a spectral analysis standpoint. Our investigation emphasizes the importance of a comprehensive understanding and acknowledgment of the subtleties involved in the computation of PCA. As such, we believe this paper offers a crucial contribution to the nuanced understanding of this foundational statistical method and stands as a valuable addition to the academic literature in the field of statistics.

Motivation & Objective

• To investigate the conditions under which SVD-based PCA without centering produces embeddings aligned with those from centered PCA.
• To clarify the role of the first right singular vector in capturing the mean direction of the data.
• To provide spectral-level theoretical bounds on the error introduced by omitting centering in SVD-based PCA.
• To challenge and formalize the common heuristic that the second to (k+1)-th singular vectors of an uncentered matrix can approximate the first k principal components.
• To enhance understanding of the algorithmic foundations of SVD-based PCA, particularly in high-dimensional settings.

Proposed method

• The authors compare SVD of the centered data matrix $\bar{X} = X - \mathbf{1}_n \bar{x}^\top$ and the uncentered matrix $X$, analyzing their singular vectors and values.
• They derive a spectral bound relating the partial sums of squared singular values of $X$ and $\bar{X}$, showing that $\|X\|_{2,k+1}^2 - (\|\bar{X}\|_{2,k}^2 + n\|\bar{x}\|^2)$ is bounded by $\bar{\sigma}_1^2$.
• The analysis uses the relationship between the first right singular vector of $X$ and the mean direction $\bar{x}$, establishing a link between SVD and data centering.
• They apply matrix perturbation theory and spectral decomposition to characterize the discrepancy between embeddings from centered and uncentered SVD.
• Theoretical results are derived under general assumptions, including $p < n$, and extended to low-rank data matrices via Corollary 7.
• The paper uses a spectral interpretation to explain the folklore belief that the second to (k+1)-th right singular vectors of $X$ can approximate the first k principal components.

Experimental results

Research questions

• RQ1 Under what conditions is the PCA embedding from uncentered SVD approximately aligned with that from centered SVD?
• RQ2 How does the first right singular vector of the uncentered data matrix relate to the mean direction of the data?
• RQ3 What is the theoretical bound on the discrepancy between the total variance explained by the first k components in centered and uncentered SVD?
• RQ4 Can the folklore heuristic—using the second to (k+1)-th right singular vectors of an uncentered matrix as a substitute for the first k components of centered PCA—be theoretically justified?
• RQ5 How does the rank of the data matrix affect the accuracy of uncentered SVD as an approximation to true PCA?

Key findings

• The discrepancy between the total variance explained by the first k components in uncentered and centered SVD is bounded by the square of the largest singular value of the centered matrix, $\bar{\sigma}_1^2$, as shown in Theorem 6.
• When the data matrix is low-rank, the discrepancy is still bounded by $\bar{\sigma}_1^2$, independent of the true rank, as formalized in Corollary 7.
• The first right singular vector of the uncentered matrix $X$ is aligned with the mean direction $\bar{x}$, explaining why it captures the global location of the data.
• The SVD of the uncentered matrix $X$ can be spectrally related to that of $\bar{X}$, with the difference in partial sums of squared singular values bounded by $\bar{\sigma}_1^2$, indicating that uncentered SVD captures most of the principal component structure.
• The bound in Theorem 6 shows that $\|X\|_{2,k+1}^2 - (\|\bar{X}\|_{2,k}^2 + n\|\bar{x}\|^2)$ lies within $(-\bar{\sigma}_1^2, \bar{\sigma}_1^2)$, providing a quantitative measure of approximation error.
• The results validate the folklore heuristic in restricted settings, showing that using the second to (k+1)-th right singular vectors of $X$ can yield a good approximation to the first k principal components of $\bar{X}$, especially when $\bar{\sigma}_1^2$ is small relative to the total variance.