[Paper Review] Bi-fidelity Variational Auto-encoder for Uncertainty Quantification

Nuojin Cheng, Osman Asif Malik+3 more|arXiv (Cornell University)|May 25, 2023

Probabilistic and Robust Engineering Design51 references4 citations

TL;DR

This paper proposes a bi-fidelity variational auto-encoder (BF-VAE) that leverages low-fidelity (LF) and high-fidelity (HF) data to significantly reduce the number of HF samples needed for accurate uncertainty quantification (UQ) in physical systems. By integrating bi-fidelity auto-regression in the latent space and optimizing a variational lower bound with limited HF data, BF-VAE achieves superior statistical accuracy over HF-only VAEs when HF data is scarce, with KID scores improving by up to 40% at low HF sample counts.

ABSTRACT

Quantifying the uncertainty of quantities of interest (QoIs) from physical systems is a primary objective in model validation. However, achieving this goal entails balancing the need for computational efficiency with the requirement for numerical accuracy. To address this trade-off, we propose a novel bi-fidelity formulation of variational auto-encoders (BF-VAE) designed to estimate the uncertainty associated with a QoI from low-fidelity (LF) and high-fidelity (HF) samples of the QoI. This model allows for the approximation of the statistics of the HF QoI by leveraging information derived from its LF counterpart. Specifically, we design a bi-fidelity auto-regressive model in the latent space that is integrated within the VAE's probabilistic encoder-decoder structure. An effective algorithm is proposed to maximize the variational lower bound of the HF log-likelihood in the presence of limited HF data, resulting in the synthesis of HF realizations with a reduced computational cost. Additionally, we introduce the concept of the bi-fidelity information bottleneck (BF-IB) to provide an information-theoretic interpretation of the proposed BF-VAE model. Our numerical results demonstrate that BF-VAE leads to considerably improved accuracy, as compared to a VAE trained using only HF data, when limited HF data is available.

Motivation & Objective

• To address the computational bottleneck in uncertainty quantification (UQ) where high-fidelity (HF) simulations are expensive and scarce.
• To reduce the number of HF realizations required to train generative models for accurate QoI statistics estimation.
• To develop a deep generative model that transfers knowledge from LF to HF data while preserving statistical fidelity in the latent space.
• To provide an information-theoretic foundation for the model using a bi-fidelity information bottleneck (BF-IB) framework.
• To demonstrate that the proposed BF-VAE outperforms HF-only VAEs in low-data regimes, especially when HF data is limited.

Proposed method

• The BF-VAE employs a probabilistic encoder-decoder architecture where the latent space is shared between LF and HF data, enabling joint modeling.
• A bi-fidelity auto-regressive structure is introduced in the latent space to model dependencies between LF and HF representations.
• The model maximizes a variational lower bound of the HF log-likelihood using a hybrid loss that incorporates both LF and HF data, even with limited HF samples.
• The training algorithm uses a two-stage process: first pre-training on LF data, then joint fine-tuning with HF data to minimize reconstruction and KL divergence terms.
• An information-theoretic interpretation is provided via the bi-fidelity information bottleneck (BF-IB), which quantifies the trade-off between fidelity and compression in the latent representation.
• The model uses fully connected neural networks with GeLU activations and a 4-dimensional latent space, trained via Adam optimizer with a fixed β hyperparameter.

Experimental results

Research questions

• RQ1 Can a bi-fidelity VAE architecture reduce the number of required high-fidelity simulations while maintaining or improving statistical accuracy in uncertainty quantification?
• RQ2 How does the integration of low-fidelity data improve the generalization and sample quality of a VAE trained on limited high-fidelity data?
• RQ3 What is the information-theoretic role of the latent space in balancing fidelity and compression across low- and high-fidelity data?
• RQ4 Does the proposed BF-VAE outperform a standard HF-only VAE in terms of distributional similarity when HF data is scarce?
• RQ5 To what extent does the bi-fidelity auto-regressive structure in the latent space enhance the model’s ability to generate realistic HF realizations?

Key findings

• BF-VAE achieves significantly better KID scores than HF-only VAEs when the number of HF realizations is small, with improvements up to 40% at n=10 HF samples.
• At low HF sample counts (n < 100), BF-VAE consistently outperforms HF-VAE in distributional similarity, as measured by the Kullback–Leibler divergence (KID).
• The model reduces the effective HF training cost to approximately 4.08 HF realizations when using 400 LF samples, due to the LF data’s contribution to the training objective.
• When HF data is abundant (n > 600), the performance of HF-VAE surpasses BF-VAE, confirming the expected behavior of multi-fidelity models in high-data regimes.
• Visual comparisons show that BF-VAE-generated samples closely match the true HF distribution, especially in capturing tail behaviors and extreme events.
• The bi-fidelity information bottleneck (BF-IB) provides a theoretical justification for the model’s ability to extract relevant, compressed representations from LF data that inform the HF distribution.