[Paper Review] Challenges in Training PINNs: A Loss Landscape Perspective

Pratik Rathore, Weimu Lei+3 more|arXiv (Cornell University)|Feb 2, 2024

Biomedical and Engineering Education37 citations

TL;DR

The paper analyzes why Physics-Informed Neural Networks (PINNs) are hard to train due to ill-conditioned loss landscapes caused by differential operators, and shows that combining first- and second-order optimization (Adam+L-BFGS) and a new second-order method (NysNewton-CG) yields significant performance gains, supported by theory and experiments.

ABSTRACT

This paper explores challenges in training Physics-Informed Neural Networks (PINNs), emphasizing the role of the loss landscape in the training process. We examine difficulties in minimizing the PINN loss function, particularly due to ill-conditioning caused by differential operators in the residual term. We compare gradient-based optimizers Adam, L-BFGS, and their combination Adam+L-BFGS, showing the superiority of Adam+L-BFGS, and introduce a novel second-order optimizer, NysNewton-CG (NNCG), which significantly improves PINN performance. Theoretically, our work elucidates the connection between ill-conditioned differential operators and ill-conditioning in the PINN loss and shows the benefits of combining first- and second-order optimization methods. Our work presents valuable insights and more powerful optimization strategies for training PINNs, which could improve the utility of PINNs for solving difficult partial differential equations.

Motivation & Objective

• Investigate why PINN Loss L is hard to minimize due to ill-conditioning from differential operators in the residual term.
• Empirically compare Adam, L-BFGS, and Adam+L-BFGS across PDEs to identify effective training strategies.
• Develop and evaluate a novel second-order optimizer (NysNewton-CG) to improve PINN performance.
• Provide theoretical justification for why combining first- and second-order methods accelerates convergence.
• Demonstrate that achieving near-zero loss is crucial for accurate PINN solutions.

Proposed method

• Analyze the PINN loss landscape by examining the Hessian spectrum before and after preconditioning.
• Compare optimizers (Adam, L-BFGS, Adam+L-BFGS) across convection, wave, and reaction PDEs with varying network widths.
• Introduce NysNewton-CG (NNCG), a Nyström-preconditioned conjugate gradient method to solve the Newton step.
• Theoretically connect ill-conditioned differential operators to ill-conditioning of the PINN loss (Theorem 8.4 informal).
• Demonstrate that a damped Newton phase can achieve high-precision solutions (Algorithm 1 GDND) and justify using Adam+L-BFGS before NNCG.

Figure 1: On the wave PDE, Adam converges slowly due to ill-conditioning and the combined Adam+L-BFGS optimizer stalls after about 40000 steps. Running NNCG (our method) after Adam+L-BFGS provides further improvement.

Experimental results

Research questions

• RQ1 Does the PINN loss exhibit ill-conditioning due to differential operators in the residual term?
• RQ2 Do optimization strategies that combine first- and second-order methods outperform purely first- or second-order approaches for PINNs?
• RQ3 Can a novel second-order method (NysNewton-CG) substantially improve PINN accuracy beyond Adam+L-BFGS?
• RQ4 How does preconditioning affect the Hessian spectrum and convergence speed in PINN training?
• RQ5 Is near-zero training loss necessary to achieve low L2 relative error in PINNs?

Key findings

• The PINN loss is ill-conditioned, with large outlier Hessian eigenvalues and significant mass near zero across convection, reaction, and wave PDEs.
• L-BFGS preconditioning reduces Hessian eigenvalues and condition numbers by at least 10^3 across all problems.
• Adam+L-BFGS consistently achieves smaller final loss and L2 relative error than Adam or L-BFGS alone across network widths and PDEs.
• A novel second-order method, NysNewton-CG (NNCG), after Adam+L-BFGS, further reduces loss and gradient norms and improves L2 relative error.
• Theoretical results show ill-conditioned differential operators lead to ill-conditioned PINN losses; combining first- and second-order methods enhances convergence.
• A damped Newton phase (GDND) can achieve fast linear convergence independent of the condition number, supporting the practical benefit of hybrid optimization.

Figure 2: We plot the final L2RE against the final loss for each combination of network width, optimization strategy, and random seed. Across all three PDEs, a lower loss generally corresponds to a lower L2RE.