[Paper Review] Deep Gaussian Processes for Multi-fidelity Modeling

Kurt Cutajar, Mark Pullin+3 more|arXiv (Cornell University)|Mar 18, 2019

Gaussian Processes and Bayesian Inference31 references41 citations

TL;DR

Introduces mf-dgp, a deep Gaussian process model for multi-fidelity data that enables end-to-end training and uncertainty propagation across fidelity levels, improving uncertainty calibration and decision-making in active learning and experimental design.

ABSTRACT

Multi-fidelity methods are prominently used when cheaply-obtained, but possibly biased and noisy, observations must be effectively combined with limited or expensive true data in order to construct reliable models. This arises in both fundamental machine learning procedures such as Bayesian optimization, as well as more practical science and engineering applications. In this paper we develop a novel multi-fidelity model which treats layers of a deep Gaussian process as fidelity levels, and uses a variational inference scheme to propagate uncertainty across them. This allows for capturing nonlinear correlations between fidelities with lower risk of overfitting than existing methods exploiting compositional structure, which are conversely burdened by structural assumptions and constraints. We show that the proposed approach makes substantial improvements in quantifying and propagating uncertainty in multi-fidelity set-ups, which in turn improves their effectiveness in decision making pipelines.

Motivation & Objective

• Motivate the need for reliable multi-fidelity modeling when high-fidelity data are scarce and expensive.
• Develop a deep Gaussian process framework that can jointly model multiple fidelities.
• Provide end-to-end training with uncertainty propagation across fidelity layers to avoid overfitting.
• Demonstrate improved uncertainty quantification and practical utility in real-world and synthetic benchmarks.

Proposed method

• Model each fidelity as a layer in a deep Gaussian process, enabling composition of GPs across fidelities.
• Use a sparse variational approximation with inducing points at each layer to achieve tractable inference.
• Define a multi-fidelity kernel that combines nonlinear fidelity mappings and original inputs (Equation 9–11).
• Adopt a unified variational objective (ELBO) that sums over fidelities and layers with KL regularization (Equation 7).
• Train with stochastic variational inference, employing a two-stage optimization to stabilize early training.
• Predict outputs at any fidelity by propagating inputs through preceding fidelities with MC sampling (Equation 8).

Experimental results

Research questions

• RQ1 Can a single deep GP model capture nonlinear relationships between fidelities without overfitting?
• RQ2 Does end-to-end training of mf-dgp improve uncertainty calibration compared to stacked or autoregressive models?
• RQ3 How does mf-dgp perform on synthetic multi-fidelity tasks with linear and nonlinear fidelity mappings?
• RQ4 Is mf-dgp scalable to large real-world datasets while maintaining reliable uncertainty estimates?

Key findings

• mf-dgp yields conservative and well-calibrated uncertainty in data-sparse regions.
• On synthetic benchmarks, mf-dgp outperforms ar1 and nargp in R^2, RMSE, and mnll across linear and nonlinear fidelity mappings.
• In a large real-world malaria dataset, mf-dgp with mini-batch training achieves lower RMSE (0.063) than a high-fidelity-only GP (0.096).
• mf-dgp supports experimental design via determinantal point processes using the posterior covariance as a kernel.
• End-to-end training across fidelities avoids overfitting and improves posterior uncertainty propagation compared to prior multi-fidelity approaches.