[Paper Review] Subspace-Induced Gaussian Processes

Zilong Tan, Sayan Mukherjee|arXiv (Cornell University)|Feb 21, 2018

Gaussian Processes and Bayesian Inference2 citations

TL;DR

This paper proposes Subspace-Induced Gaussian Processes (SIGP), a low-rank GP regression model that parameterizes the covariance kernel using a sufficient dimension reduction subspace of a reproducing kernel Hilbert space. By leveraging this subspace, SIGP achieves significant computational efficiency and reduced prediction variance, outperforming standard full GP even with rank-$m \leq 3$.

ABSTRACT

We present a new Gaussian process (GP) regression model where the covariance kernel is indexed or parameterized by a sufficient dimension reduction subspace of a reproducing kernel Hilbert space. The covariance kernel will be low-rank while capturing the statistical dependency of the response to the covariates, this affords significant improvement in computational efficiency as well as potential reduction in the variance of predictions. We develop a fast Expectation-Maximization algorithm for estimating the parameters of the subspace-induced Gaussian process (SIGP). Extensive results on real data show that SIGP can outperform the standard full GP even with a low rank-$m$, $m\leq 3$, inducing subspace.

Motivation & Objective

• To address the computational inefficiency of standard Gaussian process regression in high-dimensional settings.
• To reduce prediction variance in GP models without sacrificing predictive accuracy.
• To develop a scalable GP framework that maintains statistical power through dimension reduction in the kernel space.
• To enable fast inference via a novel Expectation-Maximization algorithm for parameter estimation.
• To demonstrate that low-rank subspace-induced kernels can match or exceed the performance of full-rank GP models.

Proposed method

• Parameterize the GP covariance kernel using a low-rank subspace derived via sufficient dimension reduction in a reproducing kernel Hilbert space.
• Construct a low-rank kernel matrix that captures the essential dependency between covariates and response while reducing computational cost.
• Formulate the likelihood under the SIGP model using the induced subspace to enable efficient marginal likelihood optimization.
• Develop a fast Expectation-Maximization algorithm to jointly estimate the subspace and kernel hyperparameters.
• Use the subspace to project input covariates into a lower-dimensional space before applying GP regression.
• Leverage the structure of the kernel to ensure that the resulting covariance matrix remains positive definite and computationally tractable.

Experimental results

Research questions

• RQ1 Can a low-rank kernel induced by a sufficient dimension reduction subspace maintain predictive accuracy comparable to full-rank GP models?
• RQ2 How does the subspace-induced kernel affect computational efficiency and prediction variance in GP regression?
• RQ3 Can the proposed EM-based estimation procedure reliably recover the underlying subspace and kernel parameters?
• RQ4 What is the minimal rank ($m$) required for the subspace-induced kernel to outperform standard GP models?
• RQ5 How does SIGP perform on real-world datasets with high-dimensional inputs compared to baseline GP methods?

Key findings

• SIGP achieves significant computational speedups by using a low-rank kernel structure, even with $m \leq 3$.
• The model reduces prediction variance compared to standard full GP, particularly in high-dimensional settings.
• Extensive experiments on real data show that SIGP outperforms standard GP models in terms of predictive performance.
• The proposed EM algorithm converges reliably and efficiently to high-quality parameter estimates.
• Even with minimal rank ($m=1,2,3$), SIGP captures sufficient statistical dependency to match or exceed full GP performance.
• The subspace-induced kernel effectively identifies the most relevant directions in the input space for predicting the response.