[Paper Review] Automated Augmented Conjugate Inference for Non-conjugate Gaussian Process Models

Théo Galy-Fajou, Florian Wenzel+1 more|arXiv (Cornell University)|Feb 26, 2020

Gaussian Processes and Bayesian Inference4 citations

TL;DR

This paper introduces Automated Augmented Conjugate Inference (aaci), a novel method for efficient Bayesian inference in non-conjugate Gaussian process models. By automatically constructing an auxiliary variable augmentation that renders the model conditionally conjugate, aaci enables fast, scalable stochastic variational inference with closed-form block coordinate updates and an asymptotically exact Gibbs sampler. The approach achieves up to two orders of magnitude faster inference than state-of-the-art black-box methods while maintaining competitive predictive performance.

ABSTRACT

We propose automated augmented conjugate inference, a new inference method for non-conjugate Gaussian processes (GP) models. Our method automatically constructs an auxiliary variable augmentation that renders the GP model conditionally conjugate. Building on the conjugate structure of the augmented model, we develop two inference methods. First, a fast and scalable stochastic variational inference method that uses efficient block coordinate ascent updates, which are computed in closed form. Second, an asymptotically correct Gibbs sampler that is useful for small datasets. Our experiments show that our method are up two orders of magnitude faster and more robust than existing state-of-the-art black-box methods.

Motivation & Objective

• To develop an efficient, automated inference framework for non-conjugate Gaussian process models that scales to large datasets.
• To overcome the limitations of black-box inference methods that rely on sampling or numerical quadrature and are computationally expensive.
• To provide a unified method applicable to a broad class of super-Gaussian likelihoods without manual derivation for each model.
• To achieve both high-speed variational inference and high-quality posterior sampling via a single automated framework.
• To improve sample efficiency and convergence stability compared to existing MCMC and variational methods.

Proposed method

• Automatically constructs an auxiliary variable augmentation that renders the original non-conjugate GP model conditionally conjugate.
• Leverages the conditional conjugacy to compute all complete conditional distributions in closed form.
• Implements a scalable stochastic variational inference method using efficient block coordinate ascent updates with closed-form solutions.
• Provides an exact Gibbs sampler for small datasets, offering asymptotically correct posterior samples.
• Uses inducing points and stochastic gradient descent to scale the variational inference to large datasets.
• Applies whitening transformations and kernel matrix preconditioning to improve MCMC mixing and convergence.

Experimental results

Research questions

• RQ1 Can an automated method construct conditionally conjugate augmentations for a broad class of non-conjugate GP models?
• RQ2 How does automated conjugate inference compare in speed and stability to state-of-the-art black-box variational inference and MCMC methods?
• RQ3 To what extent does the augmentation gap—difference in ELBO between augmented and original models—affect predictive performance?
• RQ4 Can the Gibbs sampler based on the augmented model achieve better effective sample size than Hamiltonian Monte Carlo while remaining faster?
• RQ5 How well does the method scale to large datasets with millions of data points?

Key findings

• The variational inference module of aaci is up to two orders of magnitude faster than state-of-the-art black-box methods while maintaining competitive predictive performance.
• The optimization process in aaci shows greater stability, with a smooth decrease in the negative ELBO, unlike the oscillatory behavior seen in baseline methods.
• The Gibbs sampler achieves significantly higher effective sample size than Hamiltonian Monte Carlo, especially for heavy-tailed likelihoods like Student-t and Laplace.
• Despite a noticeable gap in ELBO values between the augmented and original models, predictive performance on held-out test sets remains marginally the same.
• The method scales efficiently to large datasets, with training times on the SUSY dataset (5M points) being substantially faster than competing approaches.
• The intrachain correlation in the Gibbs sampler is very low, indicating high sample efficiency, and interchain correlation is near-optimal, confirming reliable convergence.