[Paper Review] Slice sampling covariance hyperparameters of latent Gaussian models

Iain Murray, Ryan P. Adams|arXiv (Cornell University)|Jun 4, 2010

Gaussian Processes and Bayesian Inference17 references117 citations

TL;DR

This paper introduces a slice sampling method for hyperparameter inference in latent Gaussian models, particularly Gaussian processes with non-Gaussian likelihoods. By leveraging a novel surrogate data representation and post-site reparameterization, the approach enables robust, tuning-free MCMC sampling that significantly improves mixing in both strong- and weak-data regimes, outperforming standard Metropolis-Hastings and existing slice samplers.

ABSTRACT

The Gaussian process (GP) is a popular way to specify dependencies between random variables in a probabilistic model. In the Bayesian framework the covariance structure can be specified using unknown hyperparameters. Integrating over these hyperparameters considers different possible explanations for the data when making predictions. This integration is often performed using Markov chain Monte Carlo (MCMC) sampling. However, with non-Gaussian observations standard hyperparameter sampling approaches require careful tuning and may converge slowly. In this paper we present a slice sampling approach that requires little tuning while mixing well in both strong- and weak-data regimes.

Motivation & Objective

• To address the challenge of slow mixing and high tuning sensitivity in MCMC hyperparameter inference for latent Gaussian models with non-Gaussian likelihoods.
• To develop a general-purpose, robust sampling method for covariance hyperparameters that requires little to no tuning.
• To improve posterior exploration in both strong-data and weak-data regimes, where standard methods often fail.
• To provide a practical, plug-and-play solution for Bayesian inference in Gaussian process models with complex likelihoods.
• To demonstrate that simple reparameterizations like whitening the prior can outperform standard conditional updates.

Proposed method

• Proposes a slice sampling approach for hyperparameters θ by introducing a surrogate data model that reparameterizes the likelihood to enable efficient sampling.
• Uses a post-site reparameterization that transforms the conditional posterior P(θ|f) into a form amenable to slice sampling with minimal tuning.
• Employs a robust representation that works across a wide range of covariance structures and likelihoods without requiring problem-specific derivations.
• Combines the surrogate data method with slice sampling to maintain detailed balance and ensure convergence to the correct posterior.
• Applies the method to both Gaussian and non-Gaussian likelihoods, including Cox process models with inhomogeneous Poisson observations.
• Uses a reparameterization that decouples the latent variables f from the hyperparameters θ, improving mixing by reducing posterior dependence.

Experimental results

Research questions

• RQ1 Can slice sampling be made robust and tuning-free for hyperparameter inference in latent Gaussian models with non-Gaussian likelihoods?
• RQ2 How does the performance of the proposed slice sampler compare to standard Metropolis-Hastings and existing slice samplers in both strong- and weak-data regimes?
• RQ3 To what extent do surrogate data and post-site reparameterizations improve MCMC mixing compared to direct conditional updates?
• RQ4 Does the proposed method maintain efficiency across diverse likelihood models, including Poisson and binary outcomes?
• RQ5 Can simple reparameterizations like whitening the prior outperform standard conditional update schemes in practice?

Key findings

• The proposed slice sampling method with surrogate data and post-site reparameterization achieves significantly better mixing than standard Metropolis-Hastings and existing slice samplers, especially in weak-data regimes.
• On the Ionosphere classification task, the surr-site method outperformed baselines by a substantial margin, with effective sample sizes up to 2–3× higher.
• For the mining disaster dataset, all proposed methods performed well, but surr-site and surr-taylor showed superior performance due to better handling of zero-count bins.
• On the redwood tree dataset with high sparsity (many zero-count bins), surr-taylor outperformed post-taylor and prior-white, indicating the advantage of approximating the site posterior.
• The post-site and surr-site methods were found to be more robust than prior-white and baseline methods, particularly in non-Gaussian settings.
• Even simple reparameterizations like whitening the prior were shown to significantly improve mixing, suggesting they should be prioritized in practice.