[Paper Review] Gaussian Process Regression Networks

Andrew Gordon Wilson, David A. Knowles+1 more|arXiv (Cornell University)|Oct 19, 2011

Gaussian Processes and Bayesian InferenceComputer Science44 references122 citations

TL;DR

Gaussian Process Regression Networks (GPRN) introduce a novel regression framework that combines Bayesian neural networks with Gaussian processes to model input-dependent signal and noise correlations, length-scales, amplitudes, and heavy-tailed predictive distributions. The method enables superior performance in multi-output and multivariate volatility modeling, outperforming eight multi-task GP models and three volatility models on benchmark datasets, including a 1000-dimensional gene expression dataset.

ABSTRACT

We introduce a new regression framework, Gaussian process regression networks (GPRN), which combines the structural properties of Bayesian neural networks with the non-parametric flexibility of Gaussian processes. This model accommodates input dependent signal and noise correlations between multiple response variables, input dependent length-scales and amplitudes, and heavy-tailed predictive distributions. We derive both efficient Markov chain Monte Carlo and variational Bayes inference procedures for this model. We apply GPRN as a multiple output regression and multivariate volatility model, demonstrating substantially improved performance over eight popular multiple output (multi-task) Gaussian process models and three multivariate volatility models on benchmark datasets, including a 1000 dimensional gene expression dataset.

Motivation & Objective

• To develop a flexible regression framework that captures input-dependent signal and noise correlations across multiple outputs.
• To integrate the structural adaptability of Bayesian neural networks with the nonparametric expressiveness of Gaussian processes.
• To enable modeling of input-dependent length-scales, amplitudes, and heavy-tailed predictive distributions without numerical instability.
• To provide efficient inference via both Markov Chain Monte Carlo (MCMC) and variational Bayes (VB) methods.
• To demonstrate superior predictive performance on real-world multi-output and multivariate volatility datasets compared to existing models.

Proposed method

• GPRN models outputs as a linear transformation of latent Gaussian processes via a weight matrix W(x), where W(x) is itself a set of independent Gaussian processes.
• The model incorporates input-dependent signal and noise correlations through W(x) and additive noise terms σ_f and σ_y, allowing for flexible, nonstationary covariance structures.
• The joint distribution of outputs is derived as a hierarchical Gaussian process model, with predictive distributions computed via marginalization over latent functions and weights.
• Efficient MCMC inference is performed using Gibbs sampling with conditional conjugacy, while variational Bayes employs mean-field approximations with analytical updates for hyperparameters.
• The method supports non-Gaussian noise by allowing the noise distribution to be Student-t or Laplace, enhancing robustness to outliers.
• Predictive distributions are computed analytically using expectations over the variational posterior, enabling fast and stable inference.

Experimental results

Research questions

• RQ1 Can a hybrid model combining Bayesian neural networks and Gaussian processes effectively model input-dependent signal and noise correlations in multi-output regression?
• RQ2 How does GPRN perform compared to existing multi-task Gaussian process models on real-world datasets with complex correlation structures?
• RQ3 Can GPRN capture input-dependent length-scales and amplitudes while maintaining computational efficiency and numerical stability?
• RQ4 Does the inclusion of heavy-tailed noise distributions improve robustness to outliers in predictive modeling?
• RQ5 How do MCMC and variational Bayes inference procedures compare in terms of accuracy and scalability on high-dimensional regression tasks?

Key findings

• GPRN significantly outperformed eight popular multi-task Gaussian process models on gene expression and geostatistics datasets, including a 1000-dimensional gene expression dataset.
• On financial benchmark datasets, GPRN achieved substantially better performance than three multivariate volatility models, demonstrating strong predictive accuracy.
• The model successfully captured input-dependent signal and noise correlations, enabling improved predictions in scenarios with correlated response variables.
• Both MCMC and variational Bayes inference procedures were effective, with MCMC showing better numerical stability when noise was exponentiated in multivariate volatility experiments.
• The use of heavy-tailed noise distributions (e.g., Student-t) enhanced robustness to outliers, particularly in financial data with heavy-tailed error structures.
• Exponentiating the weights W(x) improved numerical stability in MCMC but did not significantly boost empirical performance, suggesting the model is inherently robust to multimodality.