[Paper Review] Multi-Objective Loss Balancing for Physics-Informed Deep Learning

Rafael Bischof, Michael Kraus|arXiv (Cornell University)|Oct 19, 2021

Model Reduction and Neural Networks63 references58 citations

TL;DR

The paper analyzes how to balance multiple loss terms in Physics-Informed Neural Networks (PINNs) and introduces a new self-adaptive method, ReLoBRaLo, to improve accuracy and efficiency over existing loss-scaling approaches. It assesses forward and inverse problems across Burgers’, Kirchhoff, and Helmholtz PDEs.

ABSTRACT

Physics-Informed Neural Networks (PINN) are algorithms from deep learning leveraging physical laws by including partial differential equations together with a respective set of boundary and initial conditions as penalty terms into their loss function. In this work, we observe the significant role of correctly weighting the combination of multiple competitive loss functions for training PINNs effectively. To this end, we implement and evaluate different methods aiming at balancing the contributions of multiple terms of the PINNs loss function and their gradients. After reviewing of three existing loss scaling approaches (Learning Rate Annealing, GradNorm and SoftAdapt), we propose a novel self-adaptive loss balancing scheme for PINNs named \emph{ReLoBRaLo} (Relative Loss Balancing with Random Lookback). We extensively evaluate the performance of the aforementioned balancing schemes by solving both forward as well as inverse problems on three benchmark PDEs for PINNs: Burgers' equation, Kirchhoff's plate bending equation and Helmholtz's equation. The results show that ReLoBRaLo is able to consistently outperform the baseline of existing scaling methods in terms of accuracy, while also inducing significantly less computational overhead.

Motivation & Objective

• Motivate the need to balance multiple loss terms arising from physical laws in PINNs.
• Evaluate existing loss-balancing schemes and identify their limitations.
• Propose and validate a novel self-adaptive loss-balancing method (ReLoBRaLo) for PINNs.
• Demonstrate performance gains on forward and inverse PINN problems across multiple PDEs.

Proposed method

• Review existing loss-scaling methods (LRAnnealing, GradNorm, SoftAdapt) and their limitations.
• Derive and present ReLoBRaLo, a self-adaptive loss-balancing scheme using relative loss progress with a random lookback (saudade).
• Balance multiple PINN loss terms via a bounded softmax-scaled set of coefficients.
• Use Bayesian optimization and grid search for hyperparameter tuning of network architecture and training settings.
• Evaluate on forward and inverse problems for Burgers’, Kirchhoff plate bending, and Helmholtz equations.

Figure 1 : Schematic of a Physics-Informed Neural Network (PINN): A fully-connected feed-forward neural network with space and time coordinates $(\mathbf{x},t)$ as inputs, approximating a solution $\hat{\mathbf{u}}(\mathbf{x},t)$ . Derivatives of $\mathbf{u}$ w.r.t. inputs are computed by automatic

Experimental results

Research questions

• RQ1 How do different loss-balancing schemes affect PINN training stability and accuracy across forward and inverse problems?
• RQ2 Can a self-adaptive loss-balancing method outperform existing approaches in PINNs with multi-term losses?
• RQ3 What is the impact of loss-term scaling on computational efficiency and convergence for Burgers’, Kirchhoff, and Helmholtz PDEs?
• RQ4 How should hyperparameters be tuned to maximize performance of adaptive loss-balancing in PINNs?

Key findings

• ReLoBRaLo consistently outperforms baseline loss-scaling methods in accuracy.
• ReLoBRaLo induces significantly less computational overhead compared to some existing methods.
• The approach uses a bounded softmax, relative loss progress, and random lookbacks to stabilize training.
• Experiments cover forward and inverse PINN problems on Burgers’, Kirchhoff plate bending, and Helmholtz equations.

Figure 2 : Burgers’ equation problem: (a) FEM reference solution, (b) PINNs results predicted with a fully-connected network consisting of two layers and 128 nodes each, and (c) squared error.