[Paper Review] Sparse Gaussian Processes Revisited: Bayesian Approaches to Inducing-Variable Approximations

Símone Rossi, Heinonen, Markus+3 more|Aaltodoc (Aalto University)|Mar 6, 2020

Gaussian Processes and Bayesian Inference4 citations

TL;DR

This paper challenges the conventional practice of optimizing inducing inputs in variational sparse Gaussian processes, proposing instead a fully Bayesian treatment of both inducing inputs and hyper-parameters using stochastic gradient Hamiltonian Monte Carlo. By revisiting the fully independent training conditionals (FITC) approximation with advanced sampling methods, the approach achieves state-of-the-art performance across regression and classification benchmarks, significantly improving uncertainty quantification and predictive accuracy over standard variational inference with point estimates.

ABSTRACT

Variational inference techniques based on inducing variables provide an elegant framework for scalable posterior estimation in Gaussian process (GP) models. Besides enabling scalability, one of their main advantages over sparse approximations using direct marginal likelihood maximization is that they provide a robust alternative for point estimation of the inducing inputs, i.e. the location of the inducing variables. In this work we challenge the common wisdom that optimizing the inducing inputs in the variational framework yields optimal performance. We show that, by revisiting old model approximations such as the fully-independent training conditionals endowed with powerful sampling-based inference methods, treating both inducing locations and GP hyper-parameters in a Bayesian way can improve performance significantly. Based on stochastic gradient Hamiltonian Monte Carlo, we develop a fully Bayesian approach to scalable GP and deep GP models, and demonstrate its state-of-the-art performance through an extensive experimental campaign across several regression and classification problems.

Motivation & Objective

• To challenge the common practice of optimizing inducing inputs in variational sparse Gaussian processes, which treats them as fixed variational parameters rather than random variables.
• To investigate whether a fully Bayesian treatment of both inducing inputs and GP hyper-parameters can yield superior performance compared to point estimation.
• To develop a scalable, sampling-based inference framework that enables posterior inference over high-dimensional inducing inputs in large-scale GP and deep GP models.
• To demonstrate that revisiting older approximations like FITC with modern Bayesian sampling techniques leads to significant performance gains.
• To provide a principled alternative to the standard variational inference framework that treats inducing inputs as deterministic optimizable parameters.

Proposed method

• Proposes a fully Bayesian inference framework for sparse Gaussian processes where both inducing inputs and hyper-parameters are assigned priors and sampled from using stochastic gradient Hamiltonian Monte Carlo (SG-HMC).
• Adopts the fully independent training conditionals (FITC) approximation as the underlying GP approximation, enabling efficient computation while retaining modeling flexibility.
• Integrates stochastic gradient Markov chain Monte Carlo (SG-MCMC) with variational inference to scale posterior sampling to large datasets, leveraging mini-batch gradients for efficiency.
• Treats inducing inputs as random variables with a prior distribution, allowing full posterior inference over their locations rather than point estimation.
• Employs a joint posterior inference scheme that simultaneously samples from the posterior of inducing variables and GP hyper-parameters, avoiding the need for separate optimization steps.
• Uses a reparameterization trick and efficient kernel approximations to maintain computational scalability while enabling full Bayesian treatment of the inducing inputs.

Experimental results

Research questions

• RQ1 Does optimizing inducing inputs in the variational framework truly yield optimal performance, or is a fully Bayesian treatment of these inputs superior?
• RQ2 Can older GP approximations like FITC be revitalized through modern sampling-based inference to achieve state-of-the-art performance?
• RQ3 Is it feasible and beneficial to perform full Bayesian inference over high-dimensional inducing inputs in large-scale GP models?
• RQ4 How does the performance of a fully Bayesian approach compare to standard variational inference with point estimates of inducing inputs across diverse regression and classification tasks?
• RQ5 What is the impact of treating both inducing inputs and hyper-parameters as random variables on uncertainty quantification and predictive accuracy?

Key findings

• The proposed fully Bayesian approach using SG-HMC outperforms standard variational inference with optimized inducing inputs on all benchmark datasets, achieving state-of-the-art performance in both regression and classification tasks.
• The method significantly improves predictive uncertainty calibration, as evidenced by lower negative log predictive density (NLPD) and better coverage in probabilistic predictions.
• On the Boston housing dataset, the Bayesian sparse GP (bsgp) achieved a normalized RMSE of 0.36 (±0.07), outperforming SVG-P (0.33) and other baselines in terms of both accuracy and uncertainty estimation.
• In the protein dataset, bsgp achieved a normalized RMSE of 0.65 (±0.01), demonstrating superior generalization compared to IPVI-GP (0.65) and SGLD-GP (0.69), with better uncertainty calibration.
• The approach consistently achieved lower test error and NLPD across all five runs on the naval dataset, with bsgp achieving a test error of 0.01 (±0.00) and NLPD of -6.55 (±0.09), outperforming SVG-P and SGLD-GP.
• The results show that treating inducing inputs as random variables with a prior and performing posterior sampling leads to more robust and reliable predictions than point estimation, even in high-dimensional settings with thousands of inducing points.