[Paper Review] Reliable extrapolation of deep neural operators informed by physics or sparse observations

Min Zhu, Handi Zhang+3 more|arXiv (Cornell University)|Dec 13, 2022

Model Reduction and Neural Networks4 citations

TL;DR

This paper proposes a framework to enable reliable extrapolation in deep neural operators like DeepONet by integrating physical laws or sparse observations through fine-tuning or multifidelity learning. It introduces a Wasserstein-based measure of extrapolation complexity and demonstrates that physics-informed or observation-informed methods significantly reduce out-of-distribution errors across multiple parametric PDEs, achieving errors below 10^{-6} in key benchmarks.

ABSTRACT

Deep neural operators can learn nonlinear mappings between infinite-dimensional function spaces via deep neural networks. As promising surrogate solvers of partial differential equations (PDEs) for real-time prediction, deep neural operators such as deep operator networks (DeepONets) provide a new simulation paradigm in science and engineering. Pure data-driven neural operators and deep learning models, in general, are usually limited to interpolation scenarios, where new predictions utilize inputs within the support of the training set. However, in the inference stage of real-world applications, the input may lie outside the support, i.e., extrapolation is required, which may result to large errors and unavoidable failure of deep learning models. Here, we address this challenge of extrapolation for deep neural operators. First, we systematically investigate the extrapolation behavior of DeepONets by quantifying the extrapolation complexity via the 2-Wasserstein distance between two function spaces and propose a new behavior of bias-variance trade-off for extrapolation with respect to model capacity. Subsequently, we develop a complete workflow, including extrapolation determination, and we propose five reliable learning methods that guarantee a safe prediction under extrapolation by requiring additional information -- the governing PDEs of the system or sparse new observations. The proposed methods are based on either fine-tuning a pre-trained DeepONet or multifidelity learning. We demonstrate the effectiveness of the proposed framework for various types of parametric PDEs. Our systematic comparisons provide practical guidelines for selecting a proper extrapolation method depending on the available information, desired accuracy, and required inference speed.

Motivation & Objective

• Address the critical limitation of deep neural operators in extrapolating beyond the training distribution, where standard models fail due to high error.
• Systematically analyze the extrapolation behavior of DeepONets by quantifying extrapolation complexity using the 2-Wasserstein distance between function spaces.
• Develop a bias-variance trade-off framework for extrapolation that links model capacity to generalization under out-of-distribution inputs.
• Propose five reliable learning methods—based on fine-tuning or multifidelity learning—that ensure safe predictions when inputs lie outside the training support.
• Provide practical guidelines for selecting the optimal method based on available data, required accuracy, and inference speed.

Proposed method

• Quantify extrapolation complexity using the 2-Wasserstein distance between the input function space (e.g., GRF with correlation length l) and the test distribution.
• Introduce a novel bias-variance trade-off analysis for extrapolation, showing that model capacity must be carefully balanced to avoid overfitting or underfitting in out-of-distribution regimes.
• Develop a fine-tuning strategy that incorporates governing PDEs as inductive biases by adding a physics-informed loss to the pre-trained DeepONet.
• Implement multifidelity Gaussian process regression (MFGPR) to combine low- and high-fidelity data, with kernel selection (e.g., Matern) adapted to problem-specific smoothness.
• Apply layer-wise locally adaptive activation functions (L-LAAF) to improve representational capacity and generalization in the trunk and branch networks.
• Use a hybrid training strategy where the DeepONet is first pre-trained on a distribution of inputs, then fine-tuned with either PDE constraints or sparse observations to improve extrapolation.

Experimental results

Research questions

• RQ1 How does the 2-Wasserstein distance between function spaces quantify the complexity of extrapolation in DeepONets?
• RQ2 What is the nature of the bias-variance trade-off in extrapolation, and how does model capacity affect generalization under out-of-distribution inputs?
• RQ3 Can physics-informed fine-tuning or sparse observation-based adaptation reliably improve extrapolation performance in DeepONets?
• RQ4 How do multifidelity learning and MFGPR compare in enhancing extrapolation robustness when high-fidelity data is limited?
• RQ5 What are the optimal hyperparameters (e.g., learning rate, kernel type, scaling factors) for reliable extrapolation across diverse PDEs?

Key findings

• The 2-Wasserstein distance provides a quantitative measure of extrapolation complexity, showing that larger distances between training and test input distributions correlate with higher prediction errors.
• Fine-tuning with physics constraints reduces the L² relative error to below 10^{-6} for the antiderivative operator and below 10^{-5} for other PDEs, even with limited high-fidelity data.
• The FT-Obs-T method, which combines fine-tuning with sparse observations, achieves stable performance across different numbers of observations and correlation lengths, with optimal λ values determined empirically.
• Multifidelity Gaussian process regression (MFGPR) with the Matern kernel (ν=1.5) significantly outperforms RBF kernels in the lid-driven cavity flow problem, reducing error by orders of magnitude.
• Layer-wise locally adaptive activation functions (L-LAAF) with tuned scaling factors (n=2 for tanh, n=10 for SiLU, etc.) improve generalization and reduce extrapolation error in complex PDEs.
• Systematic comparisons show that physics-informed fine-tuning and multifidelity learning are the most reliable methods for extrapolation, with performance depending on data availability and required accuracy.