[Paper Review] Elliptical slice sampling

Iain Murray, Ryan P. Adams+1 more|arXiv (Cornell University)|Dec 31, 2009

Gaussian Processes and Bayesian Inference9 references315 citations

TL;DR

This paper introduces elliptical slice sampling, a novel Markov chain Monte Carlo method for sampling from posterior distributions in models with multivariate Gaussian priors. It leverages the geometry of ellipses to enable efficient, parameter-free sampling by reformulating the proposal mechanism as a slice sampling procedure over an ellipse, achieving performance comparable to tuned Metropolis–Hastings methods without requiring manual step-size tuning.

ABSTRACT

Many probabilistic models introduce strong dependencies between variables using a latent multivariate Gaussian distribution or a Gaussian process. We present a new Markov chain Monte Carlo algorithm for performing inference in models with multivariate Gaussian priors. Its key properties are: 1) it has simple, generic code applicable to many models, 2) it has no free parameters, 3) it works well for a variety of Gaussian process based models. These properties make our method ideal for use while model building, removing the need to spend time deriving and tuning updates for more complex algorithms.

Motivation & Objective

• To develop a generic, robust MCMC algorithm for models with multivariate Gaussian priors that requires no manual tuning.
• To address the poor mixing of Gibbs sampling in models with strong dependencies due to correlated latent variables.
• To eliminate the need for free parameters in MCMC sampling while maintaining or improving sampling efficiency.
• To provide a drop-in replacement for existing MCMC samplers such as Gibbs or Metropolis–Hastings in Gaussian process-based models.

Proposed method

• The method formulates the proposal distribution as a move along an ellipse defined by the current state and an auxiliary draw from the prior distribution.
• It uses slice sampling principles to automatically adapt the step-size along the ellipse, ensuring detailed balance and correct stationary distribution.
• The algorithm samples from the conditional distribution over the ellipse by maintaining a bracket of acceptable states and iteratively shrinking it.
• It introduces a reparameterization that allows the use of standard slice sampling techniques on the ellipse, avoiding issues with reversibility and correctness.
• The method avoids explicit step-size parameters by treating the ellipse parameterization as a latent variable in an augmented model.
• It integrates control variables to improve mixing in low-dimensional problems, enhancing performance in high-correlation settings.

Experimental results

Research questions

• RQ1 Can a parameter-free MCMC method be designed for models with multivariate Gaussian priors that matches or exceeds the performance of tuned Metropolis–Hastings methods?
• RQ2 How can the geometry of the Gaussian prior be exploited to construct a more efficient and robust sampling mechanism than standard Gibbs or Metropolis–Hastings?
• RQ3 Can slice sampling principles be adapted to work on elliptical manifolds while preserving detailed balance and correctness of the Markov chain?
• RQ4 What is the trade-off between computational cost and sampling efficiency when using elliptical slice sampling versus traditional MCMC methods in Gaussian process models?
• RQ5 How does the method scale in high-dimensional settings, and in what scenarios does it outperform control variable-based approaches?

Key findings

• Elliptical slice sampling achieved more effective samples per unit time than tuned Metropolis–Hastings on real-world data, such as the mining disasters and USPS digit classification problems.
• On synthetic low-dimensional regression problems, control variable sampling outperformed all methods including elliptical slice sampling, but at higher computational cost.
• In high-dimensional settings, control variable sampling failed to converge, while elliptical slice sampling remained stable and effective.
• The method required no free parameters and outperformed standard Metropolis–Hastings with optimal step-size tuning in terms of effective sample count, despite higher runtime.
• Effective sample rates were consistently higher for elliptical slice sampling compared to line-based slice sampling, indicating better mixing.
• The algorithm demonstrated robustness across diverse GP-based models, including regression, classification, and time-series problems, with minimal implementation effort.