[论文解读] Multi-Objective Loss Balancing for Physics-Informed Deep Learning

Rafael Bischof, Michael Kraus|arXiv (Cornell University)|Oct 19, 2021

Model Reduction and Neural Networks参考文献 63被引用 58

一句话总结

本文分析如何在 Physics-Informed Neural Networks (PINNs) 中平衡多个损失项，并引入一种新的自适应方法 ReLoBRaLo，以提升相对于现有损失缩放方法的准确性和效率。它在 Burgers’、Kirchhoff 和 Helmholtz 假设方程的正向和反问题上进行评估。

ABSTRACT

Physics-Informed Neural Networks (PINN) are algorithms from deep learning leveraging physical laws by including partial differential equations together with a respective set of boundary and initial conditions as penalty terms into their loss function. In this work, we observe the significant role of correctly weighting the combination of multiple competitive loss functions for training PINNs effectively. To this end, we implement and evaluate different methods aiming at balancing the contributions of multiple terms of the PINNs loss function and their gradients. After reviewing of three existing loss scaling approaches (Learning Rate Annealing, GradNorm and SoftAdapt), we propose a novel self-adaptive loss balancing scheme for PINNs named \emph{ReLoBRaLo} (Relative Loss Balancing with Random Lookback). We extensively evaluate the performance of the aforementioned balancing schemes by solving both forward as well as inverse problems on three benchmark PDEs for PINNs: Burgers' equation, Kirchhoff's plate bending equation and Helmholtz's equation. The results show that ReLoBRaLo is able to consistently outperform the baseline of existing scaling methods in terms of accuracy, while also inducing significantly less computational overhead.

研究动机与目标

• Motivate the need to balance multiple loss terms arising from physical laws in PINNs.
• Evaluate existing loss-balancing schemes and identify their limitations.
• Propose and validate a novel self-adaptive loss-balancing method (ReLoBRaLo) for PINNs.
• Demonstrate performance gains on forward and inverse PINN problems across multiple PDEs.

提出的方法

• Review existing loss-scaling methods (LRAnnealing, GradNorm, SoftAdapt) and their limitations.
• Derive and present ReLoBRaLo, a self-adaptive loss-balancing scheme using relative loss progress with a random lookback (saudade).
• Balance multiple PINN loss terms via a bounded softmax-scaled set of coefficients.
• Use Bayesian optimization and grid search for hyperparameter tuning of network architecture and training settings.
• Evaluate on forward and inverse problems for Burgers’, Kirchhoff plate bending, and Helmholtz equations.

Figure 1 : Schematic of a Physics-Informed Neural Network (PINN): A fully-connected feed-forward neural network with space and time coordinates $(\mathbf{x},t)$ as inputs, approximating a solution $\hat{\mathbf{u}}(\mathbf{x},t)$ . Derivatives of $\mathbf{u}$ w.r.t. inputs are computed by automatic

实验结果

研究问题

• RQ1 How do different loss-balancing schemes affect PINN training stability and accuracy across forward and inverse problems?
• RQ2 Can a self-adaptive loss-balancing method outperform existing approaches in PINNs with multi-term losses?
• RQ3 What is the impact of loss-term scaling on computational efficiency and convergence for Burgers’, Kirchhoff, and Helmholtz PDEs?
• RQ4 How should hyperparameters be tuned to maximize performance of adaptive loss-balancing in PINNs?

主要发现

• ReLoBRaLo consistently outperforms baseline loss-scaling methods in accuracy.
• ReLoBRaLo induces significantly less computational overhead compared to some existing methods.
• The approach uses a bounded softmax, relative loss progress, and random lookbacks to stabilize training.
• Experiments cover forward and inverse PINN problems on Burgers’, Kirchhoff plate bending, and Helmholtz equations.

Figure 2 : Burgers’ equation problem: (a) FEM reference solution, (b) PINNs results predicted with a fully-connected network consisting of two layers and 128 nodes each, and (c) squared error.