[Paper Review] A unified scalable framework for causal sweeping strategies for Physics-Informed Neural Networks (PINNs) and their temporal decompositions

Michael Penwarden, Ameya D. Jagtap+3 more|arXiv (Cornell University)|Feb 28, 2023

Model Reduction and Neural Networks48 references9 citations

TL;DR

The paper introduces a unified framework to enforce causality in PINNs and XPINNs for time-dependent PDEs, proposing stacked-decomposition and window-sweeping methods, with transfer-learning-inspired initializations and a time-sweeping collocation algorithm to improve training speed and scalability.

ABSTRACT

Physics-informed neural networks (PINNs) as a means of solving partial differential equations (PDE) have garnered much attention in the Computational Science and Engineering (CS&E) world. However, a recent topic of interest is exploring various training (i.e., optimization) challenges - in particular, arriving at poor local minima in the optimization landscape results in a PINN approximation giving an inferior, and sometimes trivial, solution when solving forward time-dependent PDEs with no data. This problem is also found in, and in some sense more difficult, with domain decomposition strategies such as temporal decomposition using XPINNs. We furnish examples and explanations for different training challenges, their cause, and how they relate to information propagation and temporal decomposition. We then propose a new stacked-decomposition method that bridges the gap between time-marching PINNs and XPINNs. We also introduce significant computational speed-ups by using transfer learning concepts to initialize subnetworks in the domain and loss tolerance-based propagation for the subdomains. Finally, we formulate a new time-sweeping collocation point algorithm inspired by the previous PINNs causality literature, which our framework can still describe, and provides a significant computational speed-up via reduced-cost collocation point segmentation. The proposed methods form our unified framework, which overcomes training challenges in PINNs and XPINNs for time-dependent PDEs by respecting the causality in multiple forms and improving scalability by limiting the computation required per optimization iteration. Finally, we provide numerical results for these methods on baseline PDE problems for which unmodified PINNs and XPINNs struggle to train.

Motivation & Objective

• Characterize training challenges in forward PINNs and XPINNs and their relation to information propagation.
• Propose a unified causality-enforcing framework spanning time-slab and domain-decomposition methods.
• Introduce stacked-decomposition and window-sweeping approaches to bridge time-marching PINNs and XPINNs.
• Leverage transfer-learning ideas to accelerate subnet training and implement loss-tolerance-based subdomain propagation.
• Present a time-sweeping collocation point algorithm to speed up computation while preserving causality.

Proposed method

• Classify causality enforcement into hard, soft, and combined approaches to map existing methods.
• Introduce stacked-decomposition to connect XPINNs with time-marching PINNs for causal training.
• Propose window-sweeping collocation point algorithm to limit residual evaluations and enforce causality continuously.
• Incorporate transfer-learning concepts to initialize subnetworks in subdomains for faster convergence.
• Develop a loss-tolerance-based propagation mechanism across subdomains to accelerate training.
• Formulate a time-sweeping collocation strategy inspired by prior causality literature to reduce collocation costs while maintaining accuracy.

Experimental results

Research questions

• RQ1 How do information propagation dynamics and causality enforcement impact PINN and XPINN training for forward, time-dependent PDEs?
• RQ2 Can a unified framework bridge time-marching PINNs and XPINNs to overcome training challenges?
• RQ3 Do stacked-decomposition and window-sweeping techniques improve training speed, scalability, and robustness?
• RQ4 Can transfer-learning-inspired initialization and loss-tolerance propagation reduce computational cost without sacrificing accuracy?
• RQ5 How does a time-sweeping collocation point algorithm affect efficiency and causality enforcement across the domain?

Key findings

• A unified framework is proposed to address training challenges in PINNs and XPINNs by enforcing causality in multiple forms.
• Stacked-decomposition bridges time-marching PINNs and XPINNs, forming a causal XPINN framework.
• Window-sweeping collocation point algorithm reduces computational cost while enforcing causality, with significant speed-ups.
• Transfer-learning-inspired initialization substantially accelerates subdomain training.
• Loss-tolerance-based propagation across subdomains speeds up training without compromising stability.
• Numerical results demonstrate improved trainability on baseline PDE problems where unmodified PINNs and XPINNs struggle.