[Paper Review] Deep Gaussian Processes for Regression using Approximate Expectation Propagation

Thang D. Bui, Daniel Hernández-Lobato+3 more|arXiv (Cornell University)|Feb 12, 2016

Gaussian Processes and Bayesian Inference29 references135 citations

TL;DR

The paper introduces a scalable approximate Bayesian method for training Deep Gaussian Processes (DGPs) using a sparse FITC approximation, an EP-based energy optimization with tied factors, and probabilistic backpropagation, delivering superior regression performance over GP and many Bayesian NN approaches.

ABSTRACT

Deep Gaussian processes (DGPs) are multi-layer hierarchical generalisations of Gaussian processes (GPs) and are formally equivalent to neural networks with multiple, infinitely wide hidden layers. DGPs are nonparametric probabilistic models and as such are arguably more flexible, have a greater capacity to generalise, and provide better calibrated uncertainty estimates than alternative deep models. This paper develops a new approximate Bayesian learning scheme that enables DGPs to be applied to a range of medium to large scale regression problems for the first time. The new method uses an approximate Expectation Propagation procedure and a novel and efficient extension of the probabilistic backpropagation algorithm for learning. We evaluate the new method for non-linear regression on eleven real-world datasets, showing that it always outperforms GP regression and is almost always better than state-of-the-art deterministic and sampling-based approximate inference methods for Bayesian neural networks. As a by-product, this work provides a comprehensive analysis of six approximate Bayesian methods for training neural networks.

Motivation & Objective

• Motivate and enable scalable Bayesian learning for Deep Gaussian Processes (DGPs) in regression.
• Develop a new inference scheme combining FITC sparse GP, approximate EP with a tied factor constraint, and probabilistic backpropagation.
• Demonstrate improved predictive performance and uncertainty calibration on diverse real-world datasets compared to GP regression and Bayesian NN methods.

Proposed method

• Use Fully Independent Training Conditional (FITC) to sparsify the GP layers and introduce inducing outputs u to reduce cubic complexity.
• Apply an approximate Expectation Propagation (EP) energy with a tied factor constraint to infer the posterior over inducing outputs and hyperparameters.
• Employ probabilistic backpropagation (assumed density filtering) to propagate Gaussian approximations through the deep GP layers and compute log Z and gradients.
• Directly optimize the EP energy (or its tied-factor equivalent) to obtain scalable inference without a costly double-loop EP procedure.
• Enable stochastic optimization by exploiting minibatches for scalable training with an objective that aggregates per-datapoint contributions.
• Provide a probabilistic predictive distribution by a forward Gaussian propagation through layers.

Experimental results

Research questions

• RQ1 Can Deep Gaussian Processes be trained efficiently at medium to large scale using sparse approximations and EP-based inference?
• RQ2 Does the proposed FITC-DGP with approximate EP and probabilistic backpropagation improve regression performance and uncertainty quantification over standard GPs and Bayesian NNs on real-world datasets?
• RQ3 How does adding more layers, dimensionality, or inducing points affect predictive performance and uncertainty in DGPs?
• RQ4 Is the method scalable in computation and memory comparable to variational approaches while maintaining accuracy?

Key findings

• DGPs with the proposed inference scheme outperform GP regression across all tested datasets.
• In most cases, DGPs with additional layers or higher-dimensional hidden layers yield further predictive gains over baselines.
• The method often surpasses state-of-the-art deterministic and sampling-based approximate inference methods for Bayesian neural networks on the evaluated tasks.
• DGPs provide well-calibrated uncertainty estimates and competitive RMSE results, with VI-based BNN methods generally underperforming in uncertainty quantification.