[Paper Review] Marginalised Gaussian Processes with Nested Sampling

Fergus Simpson, Vidhi Lalchand+1 more|arXiv (Cornell University)|Oct 30, 2020

Gaussian Processes and Bayesian Inference30 references4 citations

TL;DR

This paper proposes using nested sampling to marginalize over kernel hyperparameters in Gaussian processes, addressing the limitations of maximum likelihood type II (ML-II) optimization that often leads to overconfident predictions and local minima. By sampling from the full posterior over hyperparameters, especially in multimodal likelihood landscapes common with spectral mixture kernels, nested sampling yields more robust and reliable prediction intervals, outperforming both ML-II and Hamiltonian Monte Carlo in uncertainty quantification.

ABSTRACT

Gaussian Process (GPs) models are a rich distribution over functions with inductive biases controlled by a kernel function. Learning occurs through the optimisation of kernel hyperparameters using the marginal likelihood as the objective. This classical approach known as Type-II maximum likelihood (ML-II) yields point estimates of the hyperparameters, and continues to be the default method for training GPs. However, this approach risks underestimating predictive uncertainty and is prone to overfitting especially when there are many hyperparameters. Furthermore, gradient based optimisation makes ML-II point estimates highly susceptible to the presence of local minima. This work presents an alternative learning procedure where the hyperparameters of the kernel function are marginalised using Nested Sampling (NS), a technique that is well suited to sample from complex, multi-modal distributions. We focus on regression tasks with the spectral mixture (SM) class of kernels and find that a principled approach to quantifying model uncertainty leads to substantial gains in predictive performance across a range of synthetic and benchmark data sets. In this context, nested sampling is also found to offer a speed advantage over Hamiltonian Monte Carlo (HMC), widely considered to be the gold-standard in MCMC based inference.

Motivation & Objective

• To address the overconfidence and local minima issues in standard ML-II training of Gaussian processes by properly accounting for hyperparameter uncertainty.
• To evaluate the effectiveness of nested sampling as a robust inference method for multimodal posterior distributions in GP hyperparameter spaces.
• To demonstrate that marginalizing over hyperparameters leads to more reliable prediction intervals, especially in high-dimensional or noisy data regimes.
• To provide a principled alternative to point estimates in GP kernel learning, particularly for expressive kernels like spectral mixture that exhibit complex likelihood geometries.

Proposed method

• The paper employs nested sampling to explore the posterior distribution over GP hyperparameters, avoiding reliance on gradient-based optimization.
• It uses the spectral mixture kernel as a flexible, expressive covariance function capable of modeling complex patterns through spectral density reparameterization.
• The method integrates over hyperparameters via nested sampling, generating a set of live points that explore multiple modes in the marginal likelihood surface.
• Prediction is performed by averaging over the full posterior distribution of hyperparameters, leading to more robust predictive distributions.
• The approach is compared against Hamiltonian Monte Carlo and ML-II optimization on time-series and 2D regression tasks using a single GPU.
• A custom prior is proposed for spectral mixture kernel hyperparameters to improve sampling efficiency and posterior exploration.

Experimental results

Research questions

• RQ1 Can nested sampling effectively sample from multimodal posterior distributions of GP hyperparameters, especially in the presence of complex kernel structures like spectral mixture?
• RQ2 How does marginalizing over hyperparameters using nested sampling compare to ML-II and HMC in terms of predictive uncertainty and interval quality?
• RQ3 Does accounting for hyperparameter uncertainty lead to more reliable and robust prediction intervals in time-series and regression tasks?
• RQ4 What are the computational trade-offs of using nested sampling versus traditional ML-II or HMC for GP inference?

Key findings

• Nested sampling produces wider and more reliable prediction intervals than ML-II, which tends to be overconfident due to point estimates of hyperparameters.
• The method successfully explores multiple modes in the marginal likelihood surface, particularly in spectral mixture kernel models with complex, multimodal geometry.
• Despite sampling from a more complex posterior, nested sampling was not substantially slower than ML-II training, with only a ~2x slowdown compared to HMC in time-series benchmarks.
• The predictive performance of the marginalized GP with nested sampling exceeded that of the standard ML-II approach, especially in noisy or data-scarce regimes.
• Hamiltonian Monte Carlo was slower than nested sampling by a factor of approximately two in the experiments, despite sampling from the same posterior.
• The results demonstrate that full marginalization over hyperparameters leads to improved robustness and reduced susceptibility to local minima and overfitting.