[Paper Review] Generalized probabilistic principal component analysis of correlated data

Mengyang Gu, Weining Shen|arXiv (Cornell University)|Aug 31, 2018

Spatial and Panel Data AnalysisEconomics, Econometrics and Finance18 citations

TL;DR

This paper proposes Generalized Probabilistic Principal Component Analysis (GPPCA), a novel extension of probabilistic PCA that models correlated multivariate outputs using Gaussian processes for latent factors. By deriving a closed-form maximum marginal likelihood estimator for factor loadings and exploiting the structure of the precision matrix, GPPCA achieves linear computational complexity and significantly improves estimation accuracy and computational efficiency over traditional PPCA and other methods in high-dimensional, correlated data settings.

ABSTRACT

Principal component analysis (PCA) is a well-established tool in machine learning and data processing. The principal axes in PCA were shown to be equivalent to the maximum marginal likelihood estimator of the factor loading matrix in a latent factor model for the observed data, assuming that the latent factors are independently distributed as standard normal distributions. However, the independence assumption may be unrealistic for many scenarios such as modeling multiple time series, spatial processes, and functional data, where the outcomes are correlated. In this paper, we introduce the generalized probabilistic principal component analysis (GPPCA) to study the latent factor model for multiple correlated outcomes, where each factor is modeled by a Gaussian process. Our method generalizes the previous probabilistic formulation of PCA (PPCA) by providing the closed-form maximum marginal likelihood estimator of the factor loadings and other parameters. Based on the explicit expression of the precision matrix in the marginal likelihood that we derived, the number of the computational operations is linear to the number of output variables. Furthermore, we also provide the closed-form expression of the marginal likelihood when other covariates are included in the mean structure. We highlight the advantage of GPPCA in terms of the practical relevance, estimation accuracy and computational convenience. Numerical studies of simulated and real data confirm the excellent finite-sample performance of the proposed approach.

Motivation & Objective

• To address the limitation of traditional probabilistic PCA (PPCA), which assumes independent latent factors, in modeling correlated multivariate outputs such as time series, spatial processes, and functional data.
• To develop a generalized probabilistic PCA (GPPCA) framework that models each latent factor as a Gaussian process, enabling flexible correlation structures across inputs.
• To derive a closed-form maximum marginal likelihood estimator for the factor loading matrix under shared or distinct covariance functions of the factor processes.
• To ensure computational efficiency by showing that the inverse of the covariance matrix has an explicit form, reducing computational complexity to linear in the number of output variables.
• To demonstrate the method’s practical relevance, estimation accuracy, and scalability through simulations and real-world temperature data analysis.

Proposed method

• GPPCA extends PPCA by modeling each column of the factor loading matrix as a realization of a Gaussian process, allowing for correlation in the latent factors across inputs.
• The method assumes orthonormal factor loading vectors for identifiability and derives a closed-form solution for the maximum marginal likelihood estimator of the factor loadings when the covariance function of the factor processes is shared.
• For distinct covariance functions, the estimation reduces to an optimization problem on the Stiefel manifold, which is solved using a fast numerical algorithm from Wen and Yin (2013).
• The marginal likelihood is derived in closed form, with the precision matrix of the output distribution having an explicit structure that enables linear-time computation.
• The method incorporates covariates in the mean structure and provides a closed-form expression for the marginal likelihood in such cases.
• Predictive distributions are computed using the derived posterior, enabling efficient prediction on held-out data with uncertainty quantification.

Experimental results

Research questions

• RQ1 Can a probabilistic PCA framework be generalized to handle correlated multivariate outputs by modeling latent factors as Gaussian processes?
• RQ2 Does the closed-form maximum marginal likelihood estimator in GPPCA maintain estimation accuracy while reducing computational complexity compared to standard PPCA and other methods?
• RQ3 How does GPPCA perform in finite samples when the true factor processes are correlated, especially in high-dimensional settings?
• RQ4 What is the impact of shared versus distinct covariance functions for the latent factor processes on estimation and prediction performance?
• RQ5 How does GPPCA compare to alternative methods such as GaSP, random forests, and matrix normal models in terms of predictive accuracy and computational scalability on real-world spatio-temporal data?

Key findings

• GPPCA achieves a linear computational complexity of O(n) in the number of output variables due to the explicit form of the precision matrix, enabling efficient computation on large-scale data.
• Numerical studies show that the average mean squared error (AvgMSE) of GPPCA decreases faster with increasing sample size than Ind GP and PP GP, suggesting favorable finite-sample convergence properties.
• In the gridded temperature data analysis, GPPCA outperforms PPCA, GaSP, random forests, and matrix normal models in predictive accuracy, especially when modeling temporal and spatial correlations.
• The method successfully captures both temporal trends and spatial correlations in the temperature anomaly data, with the predictive distribution computed using a 439×240 subset for parameter estimation and the full 369,360×1 dataset for prediction.
• The closed-form marginal likelihood and efficient optimization on the Stiefel manifold make GPPCA computationally feasible for high-dimensional outputs, even when full covariance inversion would be intractable.
• The inclusion of covariates in the mean structure is supported by a closed-form marginal likelihood, enhancing model flexibility without sacrificing computational efficiency.