[Paper Review] Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Sampling

Arka Daw, Jie Bu+3 more|arXiv (Cornell University)|Jul 5, 2022

Model Reduction and Neural Networks34 citations

TL;DR

The paper introduces R3 sampling to mitigate propagation failures in PINNs by retaining high-residual regions, resampling from the domain, and releasing resolved points, with a causal extension for time-dependent PDEs, showing improved convergence and efficiency.

ABSTRACT

Despite the success of physics-informed neural networks (PINNs) in approximating partial differential equations (PDEs), PINNs can sometimes fail to converge to the correct solution in problems involving complicated PDEs. This is reflected in several recent studies on characterizing the "failure modes" of PINNs, although a thorough understanding of the connection between PINN failure modes and sampling strategies is missing. In this paper, we provide a novel perspective of failure modes of PINNs by hypothesizing that training PINNs relies on successful "propagation" of solution from initial and/or boundary condition points to interior points. We show that PINNs with poor sampling strategies can get stuck at trivial solutions if there are propagation failures, characterized by highly imbalanced PDE residual fields. To mitigate propagation failures, we propose a novel Retain-Resample-Release sampling (R3) algorithm that can incrementally accumulate collocation points in regions of high PDE residuals with little to no computational overhead. We provide an extension of R3 sampling to respect the principle of causality while solving time-dependent PDEs. We theoretically analyze the behavior of R3 sampling and empirically demonstrate its efficacy and efficiency in comparison with baselines on a variety of PDE problems.

Motivation & Objective

• Propose a propagation-based view of PINN failures where correct solutions must propagate from boundary/initial conditions to interior points.
• Develop an adaptive sampling strategy (R3) that concentrates collocation points in high PDE residual regions to alleviate propagation barriers.
• Provide theoretical guarantees for the Retain, Resample, and Release properties of R3 and its causal extension for time-dependent PDEs.
• Empirically validate R3 and Causal R3 against baselines on multiple PDE benchmarks, demonstrating improved convergence with minimal overhead.

Proposed method

• Formalize the propagation hypothesis and diagnose propagation failures via imbalance in PDE residual fields.
• Introduce Retain-Resample-Release (R3) sampling: retain high-residual points, resample new points from the domain to maintain background coverage, and release resolved points.
• Prove three properties: Retain (accumulation of high-residual points), Resample (non-empty background sampling), Release (points drop when sufficiently minimized).
• Extend R3 to Causal R3 by incorporating a time-gate function g(t) to enforce causality in both the PDE loss and sampling.
• Optionally extend to causal sampling where residuals are weighted by g(t) to respect the temporal propagation of information.

Experimental results

Research questions

• RQ1 Can propagation failures in PINNs be diagnosed via imbalanced high-residual regions in the PDE residual field?
• RQ2 Does the R3 sampling strategy mitigate propagation failures and help PINNs converge to the correct solution with lower sample complexity?
• RQ3 Does the causal extension (Causal R3) improve training for time-dependent PDEs by enforcing causality in residuals and sampling?

Key findings

• R3 and Causal R3 consistently achieve the lowest relative L2 errors across benchmark PDEs compared with baselines.
• R3 and Causal R3 dramatically reduce errors for challenging convection PDEs (e.g., beta=50) where other methods fail to converge.
• R3 mitigates propagation failures with negligible computational overhead and without maintaining dense collocation sets.
• Theoretical results confirm Retain, Resample, and Release properties, enabling accumulation of high-residual points and periodic release as regions improve.
• Causal R3 further leverages a time-gate to progressively reveal the time domain, aligning training with physical causality.