[Paper Review] A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks

Chenxi Wu, Min Zhu+3 more|arXiv (Cornell University)|Jul 21, 2022

Model Reduction and Neural Networks43 citations

TL;DR

The paper comprehensively compares ten residual-point sampling methods for PINNs, introducing two adaptive strategies (RAD and RAR-D) and showing they outperform non-adaptive methods across forward and inverse PDE problems, based on 6000+ simulations.

ABSTRACT

Physics-informed neural networks (PINNs) have shown to be an effective tool for solving forward and inverse problems of partial differential equations (PDEs). PINNs embed the PDEs into the loss of the neural network, and this PDE loss is evaluated at a set of scattered residual points. The distribution of these points are highly important to the performance of PINNs. However, in the existing studies on PINNs, only a few simple residual point sampling methods have mainly been used. Here, we present a comprehensive study of two categories of sampling: non-adaptive uniform sampling and adaptive nonuniform sampling. We consider six uniform sampling, including (1) equispaced uniform grid, (2) uniformly random sampling, (3) Latin hypercube sampling, (4) Halton sequence, (5) Hammersley sequence, and (6) Sobol sequence. We also consider a resampling strategy for uniform sampling. To improve the sampling efficiency and the accuracy of PINNs, we propose two new residual-based adaptive sampling methods: residual-based adaptive distribution (RAD) and residual-based adaptive refinement with distribution (RAR-D), which dynamically improve the distribution of residual points based on the PDE residuals during training. Hence, we have considered a total of 10 different sampling methods, including six non-adaptive uniform sampling, uniform sampling with resampling, two proposed adaptive sampling, and an existing adaptive sampling. We extensively tested the performance of these sampling methods for four forward problems and two inverse problems in many setups. Our numerical results presented in this study are summarized from more than 6000 simulations of PINNs. We show that the proposed adaptive sampling methods of RAD and RAR-D significantly improve the accuracy of PINNs with fewer residual points. The results obtained in this study can also be used as a practical guideline in choosing sampling methods.

Motivation & Objective

• Assess how different residual-point sampling strategies affect PINN accuracy and efficiency.
• Systematically compare six non-adaptive uniform sampling methods, uniform sampling with resampling, and three adaptive sampling methods (RAR-G, RAD, RAR-D).
• Propose two new adaptive sampling methods (RAD and RAR-D) and evaluate their performance across multiple forward and inverse PDE problems.
• Provide practical guidelines for selecting sampling strategies based on problem characteristics (smooth vs. oscillatory/complex solutions).

Proposed method

• Review and categorize residual-point sampling strategies into non-adaptive uniform, uniform with resampling, and adaptive methods.
• Introduce RAD: a residual-based adaptive distribution using a PDF p(x) ∝ ε(x)^k / E[ε^k] + c.
• Introduce RAR-D: a hybrid of RAR-G and RAD that adds points according to the RAD-based PDF.
• Compare to RAR-G (greedy residual refinement) and prior PDF-based resampling methods.
• Execute extensive numerical experiments ( >6000 PINN runs) across six PDE problems (four forward, two inverse) using ten sampling methods.
• Evaluate performance using L2 relative error on outputs and, for inverse problems, parameter relative errors.

Experimental results

Research questions

• RQ1 How do different residual-point sampling strategies compare in terms of accuracy for forward PINN problems?
• RQ2 How do adaptive methods (RAR-G, RAD, RAR-D) perform relative to non-adaptive methods across PDE problems with varying solution characteristics?
• RQ3 What is the impact of the residual-point count and resampling period on PINN performance?
• RQ4 Can RAD and RAR-D provide reliable improvements for inverse PINN problems (parameter recovery) compared to non-adaptive approaches?

Key findings

• RAD consistently yields the best accuracy among the ten sampling methods across all forward and inverse problems examined.
• For PDEs with complex or steep-gradient solutions (e.g., Burgers’ and multi-scale wave equations), RAD and RAR-D markedly improve accuracy with fewer residual points.
• For smooth PDEs (e.g., diffusion), some uniform methods (e.g., Hammersley, Random-R) also perform well and sometimes achieve similar errors.
• Random-R generally outperforms other fixed uniform methods among non-adaptive strategies.
• Low-discrepancy sequences (Halton, Hammersley, Sobol) often outperform Grid, LHS, and Random when residual points are fixed, with RAD/RAR-D providing further gains in adaptive settings.
• In inverse problems, RAD and RAR-D achieve the smallest reported errors for outputs and/or parameters, outperforming Grid, Random, LHS, Halton, Hammersley, Sobol, and Random-R in many cases.