[Paper Review] Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification

Radford M. Neal|ArXiv.org|Jan 28, 1997

Gaussian Processes and Bayesian InferenceComputer Science17 references391 citations

TL;DR

This paper presents a Monte Carlo implementation of Gaussian process models for Bayesian regression and classification, using Markov chain Monte Carlo (MCMC) sampling to handle non-Gaussian likelihoods and hierarchical hyperpriors. It enables robust inference for datasets up to ~1,000 cases by modeling latent variables and hyperparameters, demonstrating improved performance on regression with heavy-tailed noise and classification tasks with automatic input relevance detection.

ABSTRACT

Gaussian processes are a natural way of defining prior distributions over functions of one or more input variables. In a simple nonparametric regression problem, where such a function gives the mean of a Gaussian distribution for an observed response, a Gaussian process model can easily be implemented using matrix computations that are feasible for datasets of up to about a thousand cases. Hyperparameters that define the covariance function of the Gaussian process can be sampled using Markov chain methods. Regression models where the noise has a t distribution and logistic or probit models for classification applications can be implemented by sampling as well for latent values underlying the observations. Software is now available that implements these methods using covariance functions with hierarchical parameterizations. Models defined in this way can discover high-level properties of the data, such as which inputs are relevant to predicting the response.

Motivation & Objective

• To extend Gaussian process models to non-Gaussian likelihoods in regression and classification using MCMC.
• To enable full Bayesian inference with hierarchical hyperpriors over covariance function parameters.
• To support practical application of Gaussian processes to datasets of up to ~1,000 cases using efficient matrix operations and MCMC sampling.
• To demonstrate the ability of the model to automatically identify relevant inputs through adaptive hyperparameters.
• To provide freely available software for research and educational use, supporting flexible covariance functions and generalized models.

Proposed method

• Uses latent variable representation for non-Gaussian likelihoods, such as t-distributed noise and probit/logistic links in classification.
• Employs Markov chain Monte Carlo (MCMC) sampling to jointly update hyperparameters, latent variables, and noise variances.
• Applies conjugate prior distributions over hyperparameters and uses Gibbs sampling for posterior inference.
• Supports covariance functions with hierarchical parameterizations, including multiple exponential components for input relevance detection.
• Implements matrix operations (e.g., Cholesky decomposition) for predictive mean and variance computation in regression.
• Uses jitter to stabilize matrix inversion when the covariance matrix is ill-conditioned.

Experimental results

Research questions

• RQ1 Can Gaussian process models be effectively extended to non-Gaussian likelihoods such as t-distributed noise and binary classification using MCMC?
• RQ2 How well can hierarchical hyperpriors on covariance function parameters automatically identify relevant input variables in regression and classification?
• RQ3 What is the computational feasibility of applying Gaussian process models to datasets of moderate size (up to ~1,000 cases) using standard hardware?
• RQ4 How does the performance of Gaussian process models with MCMC inference compare to standard models in regression with heavy-tailed noise?
• RQ5 Can the framework support flexible, hierarchical covariance functions that enable automatic model selection, such as additive decompositions?

Key findings

• The model with t-distributed noise produced more robust and accurate predictions than the Gaussian noise model on a regression dataset with outliers, as evidenced by visual inspection and proximity to the true function.
• The MCMC implementation successfully handled classification problems of moderate size, with predictions based on every fifth iteration after a burn-in of 100 iterations.
• The software demonstrated that full Bayesian inference with hierarchical hyperpriors is feasible for datasets up to ~1,000 cases, requiring up to several hours on standard hardware.
• The model automatically identified relevant inputs through hyperparameters that controlled the contribution of each input to the covariance function.
• The use of latent variables enabled exact Bayesian inference for non-Gaussian likelihoods, avoiding the need for deterministic approximations.
• The implementation supports a wide range of covariance functions with hierarchical priors, enabling flexible modeling of complex data structures.