[Paper Review] On the Stability and Convergence of Physics Informed Neural Networks

Dimitrios Gazoulis, Ioannis Gkanis+1 more|arXiv (Cornell University)|Aug 10, 2023

Model Reduction and Neural NetworksPhysics and Astronomy3 citations

TL;DR

This paper establishes a rigorous mathematical framework for the stability and convergence of Physics-Informed Neural Networks (PINNs) by leveraging coercivity of energy functionals and $γ$-convergence. It demonstrates that stable training requires discrete coercivity, showing that explicit time discretization leads to instability without strict CFL-like constraints, while implicit schemes ensure convergence under suitable approximation properties of neural network spaces.

ABSTRACT

Physics Informed Neural Networks is a numerical method which uses neural networks to approximate solutions of partial differential equations. It has received a lot of attention and is currently used in numerous physical and engineering problems. The mathematical understanding of these methods is limited, and in particular, it seems that, a consistent notion of stability is missing. Towards addressing this issue we consider model problems of partial differential equations, namely linear elliptic and parabolic PDEs. Motivated by tools of nonlinear calculus of variations we systematically show that coercivity of the energies and associated compactness provide a consistent framework for stability. For time discrete training we show that if these properties fail to hold then methods may become unstable. Furthermore, using tools of $Γ$- convergence we provide new convergence results for weak solutions by only requiring that the neural network spaces are chosen to have suitable approximation properties. While our analysis is motivated by neural network-based approximation spaces, the framework developed here is applicable to any class of discrete functions satisfying the relevant approximation properties, and hence may serve as a foundation for the broader study of variational nonlinear PDE solvers.

Motivation & Objective

• To address the lack of a consistent mathematical notion of stability in Physics-Informed Neural Networks (PINNs) for solving PDEs.
• To analyze the stability of PINNs under time-discrete training, particularly contrasting explicit and implicit time discretizations.
• To provide convergence guarantees for weak solutions of linear elliptic and parabolic PDEs using $γ$-convergence theory.
• To identify coercivity and compactness as the key mathematical conditions ensuring stable and convergent PINN approximations.
• To demonstrate that failure of coercivity leads to instability, especially in explicit time discretization schemes.

Proposed method

• Formalizing PINNs as minimizers of residual-based energy functionals in $L^2$-norm over neural network spaces.
• Applying tools from nonlinear calculus of variations, particularly coercivity and compactness, to define a new notion of stability for PINNs.
• Using $γ$-convergence to prove convergence of PINN solutions to weak solutions of PDEs under approximation properties of the neural network space.
• Analyzing time-discrete PINN formulations using implicit (IE) and explicit (EE) Euler-type time discretizations.
• Constructing recovery sequences to verify the $γ$-limit of discrete energy functionals and establish convergence of minimizers.
• Performing numerical experiments with DeepXDE to compare stability of explicit vs. implicit time discretization under varying spatial training points and time steps.

Figure 1: Explicit time discrete training. Left: time step $0.4:$ the approximate solution seems that diverge. Right: time step $0.01:$ with much smaller time step the approximate solution has stable behaviour.

Experimental results

Research questions

• RQ1 Can a consistent notion of stability be defined for Physics-Informed Neural Networks solving PDEs?
• RQ2 What role does coercivity of the energy functional play in ensuring stability of PINN approximations?
• RQ3 Why do explicit time discretization schemes in PINNs lead to instability, and under what conditions can they be stabilized?
• RQ4 Under what conditions does the minimizer of the PINN energy functional converge to the true weak solution of the PDE?
• RQ5 How do the approximation properties of the neural network space affect the convergence of PINN solutions?

Key findings

• Coercivity of the energy functional and associated compactness are necessary and sufficient conditions for stability in PINNs, providing a rigorous mathematical foundation for stability.
• Explicit time discretization in PINNs fails to satisfy coercivity, leading to instability and divergence, especially when the time step is too large or spatial training points increase.
• Implicit time discretization preserves coercivity and ensures stable convergence of PINN approximations to the true solution.
• Numerical experiments confirm that explicit PINNs diverge with large time steps (e.g., $k=0.4$) but stabilize with smaller steps (e.g., $k=0.01$), supporting the theoretical analysis.
• The entire sequence of PINN minimizers converges in $L^2(0,T;H^1(Ω))$ to the true solution $u$ when the discrete energy functionals $γ$-converge and the neural network space has suitable approximation properties.
• The paper establishes that $γ$-convergence of the energy functional is sufficient for convergence of PINN solutions to weak solutions, even without assuming uniform coercivity in the discrete setting.

Figure 2: The approximations at times $t_{n}=n(0.2)$ , $n=1,2,\dots,$ are displayed with red and the initial condition with black. Left: Explicit time discrete training with time step $0.2$ and $16$ training points. The approximate solution seems that diverge. Left: Implicit time discrete training w