[Paper Review] Convergence Analysis for the PINNs.

Yuling Jiao, Yanming Lai+4 more|arXiv (Cornell University)|Sep 4, 2021

Model Reduction and Neural Networks29 references7 citations

TL;DR

This paper establishes the first convergence analysis for Physics-Informed Neural Networks (PINNs) applied to second-order elliptic PDEs with Dirichlet boundary conditions. By decomposing error into approximation and statistical components, it derives bounds on network width, depth, and training samples using ReLU³ networks and Rademacher complexity, proving convergence with explicit generalization error control.

ABSTRACT

In recent years, physical informed neural networks (PINNs) have been shown to be a powerful tool for solving PDEs empirically. However, numerical analysis of PINNs is still missing. In this paper, we prove the convergence rate to PINNs for the second order elliptic equations with Dirichlet boundary condition, by establishing the upper bounds on the number of training samples, depth and width of the deep neural networks to achieve desired accuracy. The error of PINNs is decomposed into approximation error and statistical error, where the approximation error is given in $C^2$ norm with $\mathrm{ReLU}^{3}$ networks, the statistical error is estimated by Rademacher complexity. We derive the bound on the Rademacher complexity of the non-Lipschitz composition of gradient norm with $\mathrm{ReLU}^{3}$ network, which is of immense independent interest.

Motivation & Objective

• To close the gap in theoretical understanding of PINNs by providing a rigorous convergence analysis for second-order elliptic PDEs.
• To decompose the total error in PINNs into approximation error (in $C^2$ norm) and statistical error (via Rademacher complexity).
• To derive explicit upper bounds on network width, depth, and training sample size required to achieve a desired accuracy.
• To establish a novel bound on the Rademacher complexity of the non-Lipschitz composition of the gradient norm with ReLU³ networks, which is of independent theoretical significance.

Proposed method

• Uses $C^2$ norm error bounds for approximation error, derived via ReLU³ deep neural networks.
• Applies Rademacher complexity to quantify statistical error in the loss function optimization of PINNs.
• Analyzes the composition of the gradient norm with ReLU³ networks, deriving a non-Lipschitz Rademacher complexity bound.
• Establishes convergence by balancing approximation and statistical errors through network architecture and training sample size control.
• Employs a decomposition of the total error into two distinct components: approximation and statistical, enabling separate analysis.
• Derives explicit upper bounds on network depth and width, and on the number of training samples, to achieve a target accuracy.

Experimental results

Research questions

• RQ1 What is the convergence rate of PINNs for second-order elliptic PDEs with Dirichlet boundary conditions?
• RQ2 How do network width, depth, and training sample size affect the generalization error in PINNs?
• RQ3 Can a non-Lipschitz composition of the gradient norm with a ReLU³ network be bounded using Rademacher complexity?
• RQ4 What is the relationship between approximation error in $C^2$ norm and the generalization performance of PINNs?
• RQ5 How can the total error in PINNs be decomposed and controlled through architectural and data size parameters?

Key findings

• The paper establishes the first convergence rate analysis for PINNs in solving second-order elliptic PDEs with Dirichlet conditions.
• It provides explicit upper bounds on the number of training samples, network depth, and width required to achieve a desired accuracy.
• The approximation error is bounded in the $C^2$ norm using ReLU³ networks, ensuring smoothness of the solution approximation.
• A novel bound on the Rademacher complexity of the non-Lipschitz composition of the gradient norm with ReLU³ networks is derived, which is of independent theoretical value.
• The statistical error is controlled via Rademacher complexity, enabling generalization guarantees for the PINN training process.
• The total error is shown to converge as the network width, depth, and training sample size are appropriately scaled.