[Paper Review] Doubly Stochastic Variational Inference for Deep Gaussian Processes

Hugh Salimbeni, Marc Peter Deisenroth|arXiv (Cornell University)|May 24, 2017

Gaussian Processes and Bayesian Inference16 references50 citations

TL;DR

Introduces a doubly stochastic variational inference method for Deep Gaussian Processes (DGPs) that preserves inter-layer correlations, enabling scalable training on datasets from hundreds to billions of points and improving performance over single-layer GPs and several baselines.

ABSTRACT

Gaussian processes (GPs) are a good choice for function approximation as they are flexible, robust to over-fitting, and provide well-calibrated predictive uncertainty. Deep Gaussian processes (DGPs) are multi-layer generalisations of GPs, but inference in these models has proved challenging. Existing approaches to inference in DGP models assume approximate posteriors that force independence between the layers, and do not work well in practice. We present a doubly stochastic variational inference algorithm, which does not force independence between layers. With our method of inference we demonstrate that a DGP model can be used effectively on data ranging in size from hundreds to a billion points. We provide strong empirical evidence that our inference scheme for DGPs works well in practice in both classification and regression.

Motivation & Objective

• Motivate and address the limitations of inferring Deep Gaussian Processes (DGPs) with independence assumptions between layers.
• Develop a variational posterior that maintains correlations across layers while enabling tractable computation.
• Achieve scalability to large datasets via stochastic optimization and minibatch processing.
• Demonstrate empirical performance gains of DGPs over single-layer GPs and other baselines across regression and classification tasks.

Proposed method

• Adopt a sparse inducing point variational framework within each layer to achieve tractability, while preserving inter-layer correlations by conditioning on inducing points.
• Use a variational posterior that factors across layers but retains the exact model conditioned on the inducing points, enabling end-to-end sampling.
• Represent the inducing variables q(U^l) as Gaussians with mean m^l and covariance S^l and marginalize them analytically, yielding a fully coupled but samplable posterior over F^l.
• Employ a three-part posterior structure: (i) p(F^l|U^l; F^{l-1}, Z^{l-1}); (ii) q(U^l) = N(m^l, S^l); (iii) samples drawn via the re-parameterization trick using univariate Gaussians (unfolding through layers).
• Derive a doubly stochastic evidence lower bound that uses (a) Monte Carlo sampling from the variational posterior for inter-layer samples and (b) minibatch-based data subsampling for scalability.
• Provide prediction by sampling from the variational posterior at test inputs, approximating q(f_*^L) with a Gaussian mixture via multiple samples.

Experimental results

Research questions

• RQ1 Can a variational posterior for DGPs preserve inter-layer dependencies rather than imposing independence between layers?
• RQ2 Does maintaining inter-layer correlations while using sparse inducing points enable scalable inference for DGPs on large-scale data?
• RQ3 Do multi-layer DGPs offer empirical improvements over single-layer GPs across regression and classification benchmarks, including massive datasets?
• RQ4 What is the computational trade-off (time and memory) of the proposed doubly stochastic approach compared to existing inference methods for DGPs?

Key findings

• The proposed doubly stochastic variational inference preserves layer correlations and scales to very large data, achieving improvements over single-layer GPs on several tasks.
• On regression benchmarks, deeper DGPs (2–5 layers) often outperform the sparse GP baseline with the same or more inducing points, and sometimes recover or exceed performance of the best baselines.
• In large-scale regression and classification datasets, deeper DGPs consistently improve performance over shallow models, with notable gains on very large datasets (e.g., taxi, MNIST, HIGGS, SUSY).
• For MNIST, two- and three-layer DGPs improved test accuracy from 97.48% (single-layer) to 98.06% and 98.11%, respectively.
• The method enables training of DGPs with up to five layers and demonstrates substantial RMSE improvements over SGPs on large-scale tasks (e.g., taxi with 1B points).
• The implementation is compact (<200 lines) and integrated with GPflow, facilitating practical adoption.