[Paper Review] Deep Gaussian Process Emulation using Stochastic Imputation

Deyu Ming, Daniel Williamson+1 more|arXiv (Cornell University)|Jul 4, 2021

Gaussian Processes and Bayesian Inference31 references4 citations

TL;DR

This paper proposes Stochastic Imputation (SI), a novel inference method for Deep Gaussian Process (DGP) emulation that transforms DGPs into linked Gaussian processes by stochastically imputing latent layers. The approach enables efficient, optimization-based training and analytically tractable predictions using closed-form predictive distributions, achieving performance competitive with both doubly stochastic variational inference and fully-Bayesian methods in accuracy, uncertainty quantification, and computational efficiency.

ABSTRACT

Deep Gaussian processes (DGPs) provide a rich class of models that can better represent functions with varying regimes or sharp changes, compared to conventional GPs. In this work, we propose a novel inference method for DGPs for computer model emulation. By stochastically imputing the latent layers, our approach transforms a DGP into a linked GP: a novel emulator developed for systems of linked computer models. This transformation permits an efficient DGP training procedure that only involves optimizations of conventional GPs. In addition, predictions from DGP emulators can be made in a fast and analytically tractable manner by naturally utilizing the closed form predictive means and variances of linked GP emulators. We demonstrate the method in a series of synthetic examples and empirical applications, and show that it is a competitive candidate for DGP surrogate inference, combining efficiency that is comparable to doubly stochastic variational inference and uncertainty quantification that is comparable to the fully-Bayesian approach. A $ exttt{Python}$ package $ exttt{dgpsi}$ implementing the method is also produced and available at https://github.com/mingdeyu/DGP.

Motivation & Objective

• To address the computational and inferential challenges of Deep Gaussian Process (DGP) emulation in computer models.
• To balance the speed of optimization-based methods (e.g., DSVI) with the uncertainty quantification accuracy of fully-Bayesian MCMC approaches.
• To enable efficient and analytically tractable DGP predictions by reformulating DGPs as linked GPs through latent layer imputation.
• To support flexible DGP architectures, including input-connected structures, for improved emulation performance.
• To develop a general, extensible inference framework applicable to multi-fidelity, multi-output, and hybrid emulation scenarios.

Proposed method

• Stochastically imputes the latent layers of a DGP using Elliptical Slice Sampling (ESS), transforming the DGP into a linked GP.
• Reframes DGP training as a sequence of conventional GP optimization problems, enabling efficient inference via standard GP solvers.
• Leverages the closed-form predictive mean and variance of linked GPs for fast, analytical DGP predictions.
• Applies the Stochastic Expectation Maximization (SEM) framework to iteratively refine latent layer imputations and optimize hyperparameters.
• Supports general DGP hierarchies, including non-axially aligned and input-connected structures, through the linked GP formulation.
• Integrates with existing GP libraries via the open-source Python package dgpsi, enabling deployment across diverse emulation tasks.

Experimental results

Research questions

• RQ1 Can stochastic imputation of latent layers in DGPs enable efficient, optimization-based training while preserving uncertainty quantification accuracy?
• RQ2 How does the performance of SI compare to state-of-the-art methods like DSVI and fully-Bayesian inference in terms of predictive accuracy and computational cost?
• RQ3 To what extent does incorporating input-connected structures in DGP hierarchies improve emulation performance under the SI framework?
• RQ4 Can the linked GP reformulation of DGPs support a broader range of kernel functions and hierarchical structures than existing DGP inference methods?
• RQ5 Is the SI method scalable and generalizable across diverse emulation scenarios, including multi-fidelity and multi-output systems?

Key findings

• The SI method achieves predictive accuracy comparable to fully-Bayesian inference and computational efficiency similar to DSVI, with lower NRMSEP in synthetic and empirical examples.
• Three-layered DGP emulators trained via SI with input-connected structure outperform other configurations in balancing accuracy, uncertainty quantification, and computation time.
• SI-based emulators demonstrate well-behaved predictive standard deviations across the input space, particularly in complex problems like the Heston PDE for option Greeks.
• The method enables analytically tractable predictions using closed-form expressions derived from linked GP theory, avoiding costly sampling or approximation.
• Empirical results show that two- or three-layered DGP emulators trained with SI offer optimal trade-offs between accuracy and computational cost, even as layer count increases.
• The open-source dgpsi package enables deployment of SI across diverse emulation tasks, including extensions to stochastic models via additional likelihood layers.