[Paper Review] A note on the prediction error of principal component regression

Martin Wahl|arXiv (Cornell University)|Nov 7, 2018

Statistical Methods and Inference13 references4 citations

TL;DR

This paper provides non-asymptotic upper bounds for the prediction error of principal component regression (PCR) by relating it to the reconstruction error of principal component analysis (PCA). It shows that PCR performs nearly as well as an oracle method using population-level principal components, under mild assumptions, with error bounds depending on eigenvalue decay and sample size.

ABSTRACT

We analyse the prediction error of principal component regression (PCR) and prove non-asymptotic upper bounds for the corresponding squared risk. Under mild assumptions, we show that PCR performs as well as the oracle method obtained by replacing empirical principal components by their population counterparts. Our approach relies on upper bounds for the excess risk of principal component analysis.

Motivation & Objective

• To establish non-asymptotic upper bounds for the prediction error of principal component regression (PCR) in a functional linear model framework.
• To quantify the performance gap between PCR using empirical principal components and an ideal oracle method using population-level components.
• To link the prediction error of PCR to the excess risk of PCA, thereby transferring known bounds from PCA theory to PCR.
• To derive finite-sample risk bounds under general eigenvalue decay assumptions, including polynomial decay.
• To provide a theoretical justification for PCR's performance in high-dimensional and functional data settings.

Proposed method

• The analysis is conducted in a separable Hilbert space framework, modeling the functional linear regression problem with random design.
• The prediction error is decomposed into two components: the approximation error from truncating the eigenexpansion and the estimation error from using empirical principal components.
• The excess risk of PCA is bounded using perturbation theory and empirical process tools, particularly leveraging results from Reiß and Wahl (2017) on excess risk in PCA.
• A key technical step involves constructing a sequence of projection ranks $ r $ such that the condition $ rac{ u_r}{ u_r - u_{r+1}} \leq C r $ holds, ensuring stability in the eigenvalue gap.
• The method uses a chaining argument over dyadic projections to control the sum of squared projection errors onto empirical subspaces.
• The final bound combines the approximation error $ \lambda_{d+1}^{1+2s}\|h\|^2 $, the variance term $ \frac{\sigma^2 d}{n} $, and a remainder term $ R $ involving trace and exponential decay.

Experimental results

Research questions

• RQ1 How does the prediction error of PCR compare to that of an oracle estimator using population principal components?
• RQ2 What non-asymptotic upper bounds can be derived for the squared prediction risk of PCR under general eigenvalue decay?
• RQ3 Can the error due to using empirical versus population principal components be controlled via the excess risk of PCA?
• RQ4 What are the optimal rates of convergence for PCR under polynomial eigenvalue decay $ \lambda_j \sim j^{-\alpha} $?
• RQ5 How does the sample size $ n $ and truncation dimension $ d $ jointly affect the prediction risk in PCR?

Key findings

• Under mild assumptions, the prediction error of PCR is bounded by $ C\left(\lambda_{d+1}^{1+2s}\|h\|^2 + \frac{\sigma^2 d}{n} + R\right) $, where $ R $ captures the error from empirical PCA.
• For polynomial eigenvalue decay $ \lambda_j \sim j^{-\alpha} $, the prediction risk is bounded by $ C\left(d^{-\alpha - 2s\alpha}\|h\|^2 + \frac{d^{2-\alpha}\log(ed)}{n}\|h\|^2 + \frac{\sigma^2 d}{n}\right) $, valid when $ d^2 \log(ed) \leq cn $.
• The remainder term $ R $ decays exponentially in $ n/d^2 $, indicating fast convergence when $ d $ grows slowly with $ n $.
• The bound is sharp in the sense that it matches the oracle risk up to logarithmic and constant factors, showing PCR nearly achieves optimal performance.
• The analysis establishes that the error from using empirical components is controlled by the excess risk of PCA, which is itself bounded via perturbation and empirical process techniques.
• The results hold under general assumptions on the eigenvalue decay and sub-Gaussian design, with constants depending only on $ C_{ev}, \alpha, s $, and the sub-Gaussian norm $ C_{\psi_2} $.