[Paper Review] Thoughts on Massively Scalable Gaussian Processes

Andrew Gordon Wilson, Christoph Dann+1 more|arXiv (Cornell University)|Nov 5, 2015

Gaussian Processes and Bayesian Inference25 references76 citations

TL;DR

This paper introduces Massively Scalable Gaussian Processes (MSGP), a framework that achieves $ frac{1}{2}$-order complexity reduction by combining circulant approximations, Kronecker and Toeplitz structure exploitation, and input-space projections. It enables $ frac{1}{2}$-order inference and learning complexity ($ frac{1}{2}$-order test time predictions) on billions of data points without distributed computing or severe assumptions, significantly advancing scalable GP inference and kernel learning.

ABSTRACT

We introduce a framework and early results for massively scalable Gaussian processes (MSGP), significantly extending the KISS-GP approach of Wilson and Nickisch (2015). The MSGP framework enables the use of Gaussian processes (GPs) on billions of datapoints, without requiring distributed inference, or severe assumptions. In particular, MSGP reduces the standard $O(n^3)$ complexity of GP learning and inference to $O(n)$, and the standard $O(n^2)$ complexity per test point prediction to $O(1)$. MSGP involves 1) decomposing covariance matrices as Kronecker products of Toeplitz matrices approximated by circulant matrices. This multi-level circulant approximation allows one to unify the orthogonal computational benefits of fast Kronecker and Toeplitz approaches, and is significantly faster than either approach in isolation; 2) local kernel interpolation and inducing points to allow for arbitrarily located data inputs, and $O(1)$ test time predictions; 3) exploiting block-Toeplitz Toeplitz-block structure (BTTB), which enables fast inference and learning when multidimensional Kronecker structure is not present; and 4) projections of the input space to flexibly model correlated inputs and high dimensional data. The ability to handle many ($m \approx n$) inducing points allows for near-exact accuracy and large scale kernel learning.

Motivation & Objective

• Address the computational intractability of standard Gaussian processes on large datasets ($n > 10^5$) due to $ frac{1}{2}$-order complexity.
• Overcome limitations of inducing point methods requiring $m \ll n$, which degrade predictive accuracy and hinder kernel learning.
• Enable near-exact, $ frac{1}{2}$-order test-time predictions ($ frac{1}{2}$-order per point) without distributed inference.
• Extend KISS-GP to high-dimensional inputs ($D \gg 5$) and general multidimensional structures beyond Kronecker decompositions.
• Support scalable kernel learning via fast, accurate log determinant approximations using multi-level circulant structures.

Proposed method

• Decompose covariance matrices as Kronecker products of Toeplitz matrices approximated via circulant matrices, unifying the computational benefits of fast Kronecker and Toeplitz methods.
• Use local kernel interpolation and inducing points to enable $ frac{1}{2}$-order test-time predictions for arbitrarily located inputs.
• Exploit block-Toeplitz-Toeplitz-block (BTTB) structure to enable fast, exact inference and learning when multidimensional Kronecker structure is absent.
• Apply input-space projections using a learned $d \times D$ matrix $P$ to map high-dimensional inputs into a low-dimensional subspace, enabling scalable GP modeling.
• Optimize the projection matrix $P$ jointly with kernel hyperparameters via marginal likelihood maximization, with constraints (e.g., unit scaling) to prevent degeneracy.
• Leverage circulant approximations for fast log determinant evaluations, crucial for efficient kernel learning and marginal likelihood optimization.

Experimental results

Research questions

• RQ1 Can Gaussian process inference and learning be scaled to billions of data points with $ frac{1}{2}$-order complexity, without distributed computing or restrictive assumptions?
• RQ2 Can circulant approximations unify the benefits of Kronecker and Toeplitz structures to accelerate kernel learning and log determinant computation?
• RQ3 Can BTTB structure be exploited to enable fast, exact inference in multidimensional settings where Kronecker decomposition is not applicable?
• RQ4 Can input-space projections enable KISS-GP to model high-dimensional, non-grid-structured data with $ frac{1}{2}$-order test-time complexity?
• RQ5 Can joint optimization of projection matrices and kernel hyperparameters recover ground-truth low-dimensional subspaces while maintaining predictive accuracy at scale?

Key findings

• MSGP achieves $ frac{1}{2}$-order mean and variance predictions per test point, reducing standard GP complexity from $ frac{1}{2}$-order to $ frac{1}{2}$-order.
• The method supports near-exact inference and learning with $ frac{1}{2}$-order complexity on $n \approx 10^9$ data points, enabling large-scale kernel learning.
• Subspace reconstruction error remains low (dist $< 0.1$) up to $D = 40$, with SMAE error competitive with the true GP baseline up to $D = 40$.
• Even at $D = 100$, MSGP substantially outperforms standard exact GP on high-dimensional inputs, demonstrating robustness to input dimensionality.
• Unit-scaled projection matrices prevent degeneracy issues between $P$ and kernel hyperparameters, improving numerical stability and performance.
• Circulant approximations enable fast, accurate log determinant evaluations, accelerating marginal likelihood optimization and kernel learning in 1D and multidimensional settings.