[Paper Review] MCMC for Variationally Sparse Gaussian Processes

James Hensman, Alexander Matthews+2 more|Graduate School and Research Center in Digital Science (EURECOM)|Jun 12, 2015

Gaussian Processes and Bayesian InferenceComputer Science34 references55 citations

TL;DR

This paper proposes a hybrid Markov Chain Monte Carlo (HMC) inference scheme for Gaussian processes that combines variational inducing points with non-Gaussian posterior approximations, enabling efficient, scalable Bayesian inference. The method achieves near-exact posterior estimation over function values and hyperparameters with significantly fewer inducing points than data size, outperforming Gaussian approximations in accuracy and uncertainty calibration on real-world datasets like MNIST and synthetic problems.

ABSTRACT

Gaussian process (GP) models form a core part of probabilistic machine learning. Considerable research effort has been made into attacking three issues with GP models: how to compute efficiently when the number of data is large; how to approximate the posterior when the likelihood is not Gaussian and how to estimate covariance function parameter posteriors. This paper simultaneously addresses these, using a variational approximation to the posterior which is sparse in support of the function but otherwise free-form. The result is a Hybrid Monte-Carlo sampling scheme which allows for a non-Gaussian approximation over the function values and covariance parameters simultaneously, with efficient computations based on inducing-point sparse GPs. Code to replicate each experiment in this paper will be available shortly.

Motivation & Objective

• Address the computational intractability of exact Bayesian inference in Gaussian processes for large datasets.
• Overcome limitations of existing variational methods that assume Gaussian posteriors and point estimates for hyperparameters.
• Enable joint, non-Gaussian posterior approximation over both function values and covariance function hyperparameters.
• Develop a scalable inference framework that maintains high accuracy while reducing computational cost through sparse inducing-point approximation.
• Demonstrate the feasibility and effectiveness of MCMC in large-scale GP models, challenging the perception of MCMC as too slow for practical use.

Proposed method

• Adopt a variational inducing point framework to approximate the GP posterior, reducing the computational cost of covariance matrix inversion.
• Use a free-form variational posterior over the inducing variables and hyperparameters, avoiding restrictive Gaussian assumptions.
• Implement a Hybrid Monte Carlo (HMC) sampling scheme to jointly explore the posterior over inducing variables and hyperparameters.
• Leverage the sparse GP structure to compute full posterior expectations efficiently, avoiding full covariance matrix operations.
• Integrate the HMC sampler with automatic tuning of step size and trajectory length to improve mixing and convergence.
• Use the resulting samples to estimate predictive distributions and hyperparameter posteriors with improved uncertainty quantification.

Experimental results

Research questions

• RQ1 Can MCMC sampling be made computationally feasible for large-scale Gaussian process models using sparse inducing-point approximations?
• RQ2 Does a non-Gaussian posterior approximation over both function values and hyperparameters lead to better uncertainty quantification than standard Gaussian variational approximations?
• RQ3 How many inducing points are required to achieve near-exact Bayesian inference in GP models, and how does this compare to dataset size?
• RQ4 Can the proposed HMC scheme outperform existing variational GP methods in terms of predictive accuracy and log-likelihood on real-world benchmarks?
• RQ5 To what extent do inducing point locations adapt to decision boundaries in classification tasks, and how does this affect model performance?

Key findings

• The proposed HMC-based inference scheme achieves a test log-density of -0.064 on the MNIST dataset, improving from -0.068 using only variational approximation.
• On MNIST, the model achieved 98.04% accuracy, significantly outperforming previous GP-based approaches on the same benchmark.
• The number of required inducing points was substantially smaller than the dataset size, with 500 inducing points sufficient for MNIST (70,000 images).
• The free-form posterior approximation captured strong correlations and non-linear decision boundaries in multiclass problems, unlike Gaussian approximations which were more conservative.
• HMC and Gibbs samplers showed comparable efficiency, with HMC achieving effective sample sizes (ESS) of 1.9–5.1 and TN-ESS of 2.8×10⁻³ to 3.8×10⁻⁴ across experiments.
• Inducing points moved toward decision boundaries during optimization, indicating improved representation of complex, non-linear decision surfaces in classification tasks.