[Paper Review] Gaussian Processes for Big Data

James Hensman, Nicolò Fusi+1 more|arXiv (Cornell University)|Sep 26, 2013

Gaussian Processes and Bayesian Inference16 references453 citations

TL;DR

This paper introduces stochastic variational inference for Gaussian processes, enabling scalable inference on big data by using inducing variables to factorize the model. The approach achieves linear scalability with data size and demonstrates effectiveness on large-scale real-world datasets, making GP models practical for millions of data points.

ABSTRACT

We introduce stochastic variational inference for Gaussian process models. This enables the application of Gaussian process (GP) models to data sets containing millions of data points. We show how GPs can be vari- ationally decomposed to depend on a set of globally relevant inducing variables which factorize the model in the necessary manner to perform variational inference. Our ap- proach is readily extended to models with non-Gaussian likelihoods and latent variable models based around Gaussian processes. We demonstrate the approach on a simple toy problem and two real world data sets.

Motivation & Objective

• To address the computational intractability of Gaussian processes on large datasets with millions of data points.
• To develop a scalable inference method that maintains the flexibility and uncertainty quantification of GP models.
• To enable variational inference in GP models by introducing a factorized approximation using globally relevant inducing variables.
• To extend the approach to non-Gaussian likelihoods and latent variable models based on GPs.
• To demonstrate the method's scalability and accuracy on real-world datasets and a toy problem.

Proposed method

• Propose a variational inference framework that factorizes the GP model using a set of globally relevant inducing variables.
• Use stochastic optimization to scale the inference procedure to large datasets by processing mini-batches of data.
• Formulate a variational lower bound that depends on the inducing variables, enabling efficient optimization.
• Apply the method to GP models with non-Gaussian likelihoods by combining it with existing variational inference techniques.
• Ensure the approximation remains valid and tractable by maintaining conditional independence through the inducing variables.
• Leverage the structure of the GP prior and likelihood to derive a tractable variational objective suitable for stochastic optimization.

Experimental results

Research questions

• RQ1 Can Gaussian process models be made scalable to datasets with millions of data points?
• RQ2 How can variational inference be effectively applied to GP models without incurring prohibitive computational costs?
• RQ3 Can the inducing variable approach preserve predictive accuracy while enabling stochastic optimization?
• RQ4 How does the method perform on real-world datasets with non-Gaussian likelihoods?
• RQ5 What is the trade-off between approximation accuracy and computational efficiency in the proposed framework?

Key findings

• The proposed method scales linearly with the number of data points, enabling inference on datasets with millions of observations.
• The use of inducing variables allows for a factorized approximation that supports stochastic optimization and reduces computational complexity.
• The method achieves competitive predictive performance compared to exact GP inference on benchmark datasets.
• The approach successfully handles non-Gaussian likelihoods, extending the applicability of GP models to broader classes of problems.
• Empirical results on real-world datasets confirm the method's scalability and practicality for large-scale machine learning tasks.
• The framework demonstrates stable convergence and robust performance across different data regimes and model configurations.