[Paper Review] Characterizing possible failure modes in physics-informed neural networks

Aditi S. Krishnapriyan, Amir Gholami+3 more|arXiv (Cornell University)|Sep 2, 2021

Model Reduction and Neural Networks351 citations

TL;DR

The paper shows that vanilla physics-informed neural networks (PINNs) can fail to capture nontrivial physics in convection, reaction, and diffusion problems due to optimization difficulties from soft PDE constraints, and proposes curriculum regularization and seq2seq learning to substantially improve performance.

ABSTRACT

Recent work in scientific machine learning has developed so-called physics-informed neural network (PINN) models. The typical approach is to incorporate physical domain knowledge as soft constraints on an empirical loss function and use existing machine learning methodologies to train the model. We demonstrate that, while existing PINN methodologies can learn good models for relatively trivial problems, they can easily fail to learn relevant physical phenomena for even slightly more complex problems. In particular, we analyze several distinct situations of widespread physical interest, including learning differential equations with convection, reaction, and diffusion operators. We provide evidence that the soft regularization in PINNs, which involves PDE-based differential operators, can introduce a number of subtle problems, including making the problem more ill-conditioned. Importantly, we show that these possible failure modes are not due to the lack of expressivity in the NN architecture, but that the PINN's setup makes the loss landscape very hard to optimize. We then describe two promising solutions to address these failure modes. The first approach is to use curriculum regularization, where the PINN's loss term starts from a simple PDE regularization, and becomes progressively more complex as the NN gets trained. The second approach is to pose the problem as a sequence-to-sequence learning task, rather than learning to predict the entire space-time at once. Extensive testing shows that we can achieve up to 1-2 orders of magnitude lower error with these methods as compared to regular PINN training.

Motivation & Objective

• Assess how PINNs perform on convection, reaction, and reaction-diffusion problems with nontrivial PDE coefficients.
• Identify how soft PDE constraints affect optimization and conditioning in PINNs.
• Determine whether PINN expressivity is the bottleneck or the optimization landscape is responsible for failures.
• Propose and evaluate mitigation strategies to improve PINN training and accuracy.
• Open-source the implementation to aid reproducibility and extension by researchers.

Proposed method

• Formulate PINN training with soft PDE residuals as a loss term alongside data loss and boundary conditions.
• Analyze loss landscapes by perturbing Hessian eigenvectors to observe optimization difficulty.
• Evaluate two mitigation strategies: curriculum regularization that progressively increases PDE constraint strength, and sequence-to-sequence learning that predicts solutions over shorter time segments.
• Compare vanilla PINN, curriculum PINN, and seq2seq PINN across convection, reaction, and reaction-diffusion setups.
• Use L-BFGS optimization and a 4-layer, 50-neuron per layer network with tanh activations and collocation points on a periodic domain.

Experimental results

Research questions

• RQ1 Does vanilla PINN training accurately learn solutions for convection, reaction, and reaction-diffusion PDEs with nontrivial coefficients?
• RQ2 How does soft PDE regularization affect the loss landscape and optimization in PINNs?
• RQ3 Are observed failures due to insufficient network capacity or optimization challenges?
• RQ4 Do curriculum regularization and seq2seq learning improve PINN performance and stability?
• RQ5 Can these approaches capture sharp or diffusive features better than standard PINNs?

Key findings

• PINNs achieve good solutions only for very small PDE coefficients and fail for moderately challenging regimes.
• Increasing soft PDE regularization makes the loss landscape more complex and harder to optimize, especially at higher coefficients.
• The network capacity is not the main bottleneck; optimization with the PDE-based constraint is the core issue.
• Curriculum regularization dramatically improves errors and smoothness of the loss landscape, sometimes by nearly two orders of magnitude.
• Seq2seq learning (time-marching) reduces error by up to almost two orders of magnitude for reaction and reaction-diffusion problems compared to predicting the entire space-time at once.
• Open-source implementation provides reproducibility and extension opportunities.