[Paper Review] Physics-informed neural networks with residual/gradient-based adaptive sampling methods for solving PDEs with sharp solutions

Zhiping Mao, Xuhui Meng|arXiv (Cornell University)|Feb 16, 2023

Model Reduction and Neural Networks5 citations

TL;DR

This paper proposes physics-informed neural networks (PINNs) with residual- and gradient-based adaptive sampling methods (ASMs) to improve accuracy in solving PDEs with sharp solutions. By iteratively adding residual points in regions of high residual or high solution gradient, ASM II—using both metrics—achieves superior accuracy, stability, and convergence over standard PINNs and residual-only ASM I.

ABSTRACT

We consider solving the forward and inverse PDEs which have sharp solutions using physics-informed neural networks (PINNs) in this work. In particular, to better capture the sharpness of the solution, we propose adaptive sampling methods (ASMs) based on the residual and the gradient of the solution. We first present a residual only based ASM algorithm denoted by ASM I. In this approach, we first train the neural network by using a small number of residual points and divide the computational domain into a certain number of sub-domains, we then add new residual points in the sub-domain which has the largest mean absolute value of the residual, and those points which have largest absolute values of the residual in this sub-domain will be added as new residual points. We further develop a second type of ASM algorithm (denoted by ASM II) based on both the residual and the gradient of the solution due to the fact that only the residual may be not able to efficiently capture the sharpness of the solution. The procedure of ASM II is almost the same as that of ASM I except that in ASM II, we add new residual points which not only have large residual but also large gradient. To demonstrate the effectiveness of the present methods, we employ both ASM I and ASM II to solve a number of PDEs, including Burger equation, compressible Euler equation, Poisson equation over an L-shape domain as well as high-dimensional Poisson equation. It has been shown from the numerical results that the sharp solutions can be well approximated by using either ASM I or ASM II algorithm, and both methods deliver much more accurate solution than original PINNs with the same number of residual points. Moreover, the ASM II algorithm has better performance in terms of accuracy, efficiency and stability compared with the ASM I algorithm.

Motivation & Objective

• Address the challenge of accurately solving PDEs with sharp solutions using physics-informed neural networks (PINNs), which often struggle with poor convergence and low accuracy.
• Overcome the limitations of uniform or prior-knowledge-based residual sampling by introducing adaptive sampling strategies that dynamically refine sampling in regions of high error or steep gradients.
• Improve solution accuracy and training stability for both forward and inverse problems in complex geometries and high dimensions.
• Extend adaptive refinement to boundary conditions when sharp features occur near boundaries, enhancing overall solution fidelity.
• Demonstrate the effectiveness of combining residual and gradient information for more robust and efficient sampling in PINNs.

Proposed method

• Propose ASM I: an adaptive sampling method that adds new residual points in the subdomain with the highest mean absolute residual, selecting points with the largest residual values.
• Develop ASM II: an enhanced method that incorporates both residual and gradient magnitude, adding points with large residual and large solution gradient to better capture sharp features.
• Use domain decomposition to subdivide the computational domain into S subdomains for localized error and gradient assessment.
• Apply Gaussian quadrature to compute subdomain-integrated residual and gradient norms for adaptive refinement decisions.
• Integrate adaptive boundary sampling by adding boundary points with large solution magnitude when sharpness is near the boundary.
• Iteratively train the PINN using Adam and L-BFGS optimizers, with adaptive refinement applied at each iteration until convergence or maximum iteration count.

Figure 1: Schematic of PINNs. Yellow dots: inputs of the NN, blue dots: activation function, green dots $\tilde{u}$ : output of the neural network (which is used to approximate the solution of the PDE), orange dots: differential operator, $R$ represents the residual of the PDE, $Loss_{Data}$ represe

Experimental results

Research questions

• RQ1 Can adaptive sampling based on residual and gradient improve the accuracy of PINNs for PDEs with sharp solutions compared to uniform or residual-only sampling?
• RQ2 How does incorporating solution gradient information enhance the stability and efficiency of the adaptive sampling process in PINNs?
• RQ3 To what extent does adaptive sampling near boundaries improve solution accuracy when sharp features are located at domain boundaries?
• RQ4 How do the proposed ASMs (ASM I and ASM II) compare in convergence speed, accuracy, and robustness across different PDEs, including nonlinear and high-dimensional problems?
• RQ5 Can the proposed adaptive sampling strategy significantly reduce the number of residual points required to achieve high accuracy compared to standard PINN approaches?

Key findings

• ASM II, which uses both residual and gradient information, achieves faster convergence and significantly lower $L^∞$ error than ASM I and standard PINNs on the high-dimensional Poisson equation with $d=9$.
• For the L-shape Poisson problem, the proposed method with adaptive boundary sampling improved accuracy, stability, and efficiency compared to baseline PINN and residual-only adaptive sampling.
• On the compressible Euler equation and Burgers' equation, both ASM I and ASM II outperformed standard PINNs with the same number of residual points, with ASM II showing superior accuracy and stability.
• The use of gradient information in ASM II enables more effective detection and refinement of discontinuities and sharp layers, leading to better solution resolution.
• In high-dimensional problems, ASM II achieved better accuracy with fewer residual points than ASM I, demonstrating improved sampling efficiency.
• The adaptive refinement strategy based on both residual and gradient leads to a more stable training process, especially in problems with complex or localized sharp features.

Figure 2: Schematic for the computation of mean value of the absolute value ( $\mathcal{E}$ ) in a sub-domain. Based on the Gaussian quadrature, the $\mathcal{E}$ is calculated as $\mathcal{E}=\frac{1}{x_{c}-x_{b}}\int^{x_{c}}_{x_{b}}|R(x)|dx\approx\frac{1}{x_{c}-x_{b}}\sum^{N}_{i}\omega_{i}|R(x_{i}