[Paper Review] Improved Training of Physics-Informed Neural Networks with Model Ensembles

Katsiaryna Haitsiukevich, Alexander Ilin|arXiv (Cornell University)|Apr 11, 2022

Model Reduction and Neural Networks36 references4 citations

TL;DR

This paper proposes a physics-informed neural network (PINN) ensemble method that stabilizes training by dynamically expanding the solution domain based on ensemble agreement, avoiding reliance on pre-defined schedules. The approach improves generalization and convergence, achieving competitive performance across convection, reaction, and reaction-diffusion PDEs with stable training and uncertainty estimation via confidence intervals.

ABSTRACT

Learning the solution of partial differential equations (PDEs) with a neural network is an attractive alternative to traditional solvers due to its elegance, greater flexibility and the ease of incorporating observed data. However, training such physics-informed neural networks (PINNs) is notoriously difficult in practice since PINNs often converge to wrong solutions. In this paper, we address this problem by training an ensemble of PINNs. Our approach is motivated by the observation that individual PINN models find similar solutions in the vicinity of points with targets (e.g., observed data or initial conditions) while their solutions may substantially differ farther away from such points. Therefore, we propose to use the ensemble agreement as the criterion for gradual expansion of the solution interval, that is including new points for computing the loss derived from differential equations. Due to the flexibility of the domain expansion, our algorithm can easily incorporate measurements in arbitrary locations. In contrast to the existing PINN algorithms with time-adaptive strategies, the proposed algorithm does not need a pre-defined schedule of interval expansion and it treats time and space equally. We experimentally show that the proposed algorithm can stabilize PINN training and yield performance competitive to the recent variants of PINNs trained with time adaptation.

Motivation & Objective

• To address the common failure of PINNs to converge to correct solutions due to loss function imbalance and poor initialization.
• To eliminate dependence on pre-defined time-adaptive schedules in PINN training.
• To enable flexible, data-driven expansion of the solution domain in both time and space using ensemble confidence.
• To provide uncertainty estimates through ensemble confidence intervals.
• To improve robustness and generalization across diverse PDE systems, including convection and reaction-diffusion equations.

Proposed method

• Train an ensemble of PINNs with different random initializations to capture diverse solution behaviors.
• Use ensemble agreement (confidence) as a criterion to gradually include new collocation points in the PDE loss, expanding the solution domain.
• Dynamically select new points for PDE loss based on proximity to regions of high ensemble agreement, ensuring stable propagation from known data or initial conditions.
• Treat time and space equally in domain expansion, enabling incorporation of measurements at arbitrary locations.
• Use a threshold-based mechanism to include points within a distance Δ_PDE from already included regions, balancing exploration and stability.
• Apply LBFGS fine-tuning after initial Adam pre-training to improve final accuracy, leveraging ensemble convergence.

Experimental results

Research questions

• RQ1 Can ensemble agreement be used as a reliable criterion for expanding the solution domain in PINN training?
• RQ2 Does dynamic, confidence-based domain expansion outperform fixed-schedule time-adaptive PINN methods?
• RQ3 Can the proposed method stabilize PINN training across diverse PDE types, including convection and reaction-diffusion systems?
• RQ4 How sensitive is the ensemble PINN method to hyperparameters such as Δ_PDE?
• RQ5 Can the ensemble provide meaningful uncertainty estimates through confidence intervals?

Key findings

• The proposed ensemble method achieves stable training across all tested PDE systems, including convection, reaction, and reaction-diffusion equations, with consistent performance.
• The method outperforms vanilla PINNs with both Adam and LBFGS optimizers, especially in systems where baseline methods fail to converge.
• Compared to time-adaptive baselines like bc-PINN and SA-PINN, the ensemble method achieves competitive or superior accuracy without requiring a pre-defined expansion schedule.
• The algorithm shows low sensitivity to hyperparameter settings, particularly when Δ_PDE is appropriately chosen, indicating robustness.
• The ensemble provides confidence intervals that reflect solution uncertainty, with higher agreement in regions near initial/boundary conditions and lower agreement farther away.
• LBFGS fine-tuning after Adam pre-training significantly improves final accuracy when a good solution is found during pre-training.