[Paper Review] Scalable Lévy Process Priors for Spectral Kernel Learning

Phillip A. Jang, Andrew E. Loeb+2 more|arXiv (Cornell University)|Feb 2, 2018

Gaussian Processes and Bayesian Inference9 references3 citations

TL;DR

This paper proposes a scalable Bayesian kernel learning framework using Lévy process priors over spectral densities to model uncertainty in stationary covariance functions. By leveraging the algebraic structure of Lévy processes, the method enables O(n) training and O(1) predictions via structured kernel interpolation, automatically infers model order through sparsity-inducing priors, and achieves state-of-the-art long-range extrapolation with well-calibrated uncertainty estimates on real-world benchmarks.

ABSTRACT

Gaussian processes are rich distributions over functions, with generalization properties determined by a kernel function. When used for long-range extrapolation, predictions are particularly sensitive to the choice of kernel parameters. It is therefore critical to account for kernel uncertainty in our predictive distributions. We propose a distribution over kernels formed by modelling a spectral mixture density with a Lévy process. The resulting distribution has support for all stationary covariances--including the popular RBF, periodic, and Matérn kernels--combined with inductive biases which enable automatic and data efficient learning, long-range extrapolation, and state of the art predictive performance. The proposed model also presents an approach to spectral regularization, as the Lévy process introduces a sparsity-inducing prior over mixture components, allowing automatic selection over model order and pruning of extraneous components. We exploit the algebraic structure of the proposed process for $\mathcal{O}(n)$ training and $\mathcal{O}(1)$ predictions. We perform extrapolations having reasonable uncertainty estimates on several benchmarks, show that the proposed model can recover flexible ground truth covariances and that it is robust to errors in initialization.

Motivation & Objective

• To address the lack of uncertainty quantification in spectral mixture kernel learning, particularly for long-range extrapolation.
• To eliminate manual tuning of the number of mixture components in spectral kernels by enabling automatic inference over model order.
• To develop a scalable, fully Bayesian framework for kernel learning that supports efficient training and prediction on large datasets.
• To incorporate sparsity-inducing priors through Lévy processes to prune noisy or redundant kernel components.
• To provide a conceptually interpretable, hierarchical function-space view of kernel learning with principled uncertainty propagation.

Proposed method

• Models the spectral density of a stationary kernel as a Lévy process, enabling a nonparametric prior over spectral mixture components.
• Uses scale-location mixtures of Gaussians or Laplacians as basis functions to ensure closed-form kernel expressions and support for all stationary covariance functions.
• Employs reversible jump Markov chain Monte Carlo (RJ-MCMC) to jointly infer the number of components and their parameters, enabling trans-dimensional sampling.
• Integrates structured kernel interpolation (SKI) to achieve O(n) training and O(1) prediction complexity, scaling to large datasets.
• Applies the Lévy process prior as a sparsity-inducing regularizer, promoting automatic pruning of extraneous components.
• Forms a Bayesian model average over kernel functions, leading to non-Gaussian, heavy-tailed predictive distributions with improved uncertainty calibration.

Experimental results

Research questions

• RQ1 Can a Lévy process prior over spectral densities enable automatic inference over the number of mixture components in spectral kernel learning?
• RQ2 How does incorporating kernel uncertainty via a hierarchical Bayesian model improve long-range extrapolation performance compared to fixed-kernel approaches?
• RQ3 To what extent does the Lévy process prior act as a spectral regularizer, promoting sparsity and reducing overfitting to noise?
• RQ4 Can the proposed method scale to large datasets while maintaining predictive accuracy and uncertainty calibration?
• RQ5 How robust is the method to poor initialization, and does it outperform baseline spectral mixture models in practice?

Key findings

• The model achieves state-of-the-art predictive performance on long-range extrapolation tasks, with predictive bands that fully capture withheld test data—unlike fixed-kernel spectral mixture models that often overconfidently miss the target.
• The Lévy process prior induces sparsity in mixture components, automatically identifying and pruning extraneous components without manual tuning of model order.
• On a 100,000-point natural sound texture dataset, the method trained in under one hour on a standard CPU, demonstrating O(n) scalability via structured kernel interpolation.
• Posterior samples of the kernel function concentrate around the empirical covariance, showing effective learning and convergence to the true underlying structure.
• The method is robust to initialization, with consistent performance across different starting points, unlike spectral mixture models that are highly sensitive to initial component parameters.
• The Bayesian model average over kernels yields heavy-tailed predictive distributions, reflecting realistic uncertainty in extrapolation, especially in regions with limited data.