[论文解读] Understanding and mitigating gradient pathologies in physics-informed neural networks

Sifan Wang, Yujun Teng+1 more|arXiv (Cornell University)|Jan 13, 2020

Model Reduction and Neural Networks参考文献 49被引用 197

一句话总结

该论文识别出物理信息神经网络（PINNs）中因梯度流动刚性而产生的梯度病态，并提出自适应学习率退火方案及一种新的架构以平衡损失项，从而显著提升预测精度。

ABSTRACT

The widespread use of neural networks across different scientific domains often involves constraining them to satisfy certain symmetries, conservation laws, or other domain knowledge. Such constraints are often imposed as soft penalties during model training and effectively act as domain-specific regularizers of the empirical risk loss. Physics-informed neural networks is an example of this philosophy in which the outputs of deep neural networks are constrained to approximately satisfy a given set of partial differential equations. In this work we review recent advances in scientific machine learning with a specific focus on the effectiveness of physics-informed neural networks in predicting outcomes of physical systems and discovering hidden physics from noisy data. We will also identify and analyze a fundamental mode of failure of such approaches that is related to numerical stiffness leading to unbalanced back-propagated gradients during model training. To address this limitation we present a learning rate annealing algorithm that utilizes gradient statistics during model training to balance the interplay between different terms in composite loss functions. We also propose a novel neural network architecture that is more resilient to such gradient pathologies. Taken together, our developments provide new insights into the training of constrained neural networks and consistently improve the predictive accuracy of physics-informed neural networks by a factor of 50-100x across a range of problems in computational physics. All code and data accompanying this manuscript are publicly available at \url{https://github.com/PredictiveIntelligenceLab/GradientPathologiesPINNs}.

研究动机与目标

• 解释在训练过程中PINNs为何出现梯度病态，特别是梯度流动刚性导致的问题。
• 诊断PINN损失中数据拟合项与PDE残差项之间的不平衡。
• 提出一种自适应学习率退火算法以平衡复合损失。
• 引入一种对梯度病态更具鲁棒性的神经网络架构。
• 展示在计算物理问题中PINNs预测准确性的提升。

提出的方法

• 回顾PINNs的公式化，复合损失为 L(θ)=Lr(θ)+Σi λi Li(θ)。
• 使用简单基准（如亥姆霍兹方程与泊松方程）分析边界数据拟合项与PDE残差项之间的梯度不平衡。
• 显示梯度流动的刚性可能导致标准梯度下降的不稳定，因而需要自适应方法。
• 提出一种学习率退火算法，基于梯度统计在线调整权重λi（类似自适应优化器）。
• 引入一种新颖的全连接神经网络架构，降低刚性以改善训练稳定性。
• 给出跨计算物理问题的实证演示，显示显著的预测提升。

实验结果

研究问题

• RQ1 在训练过程中，PINNs的梯度病态和反向传播梯度失衡的原因是什么？
• RQ2 梯度刚性如何影响PINNs中基于梯度的优化的稳定性和收敛性？
• RQ3 基于梯度统计的自适应损失权重分配能否稳定PINNs的训练？
• RQ4 重新设计的网络架构是否能降低梯度刚性并提高PINNs的准确性？
• RQ5 所提方法在跨越计算物理问题上的预测改进程度有多大？

主要发现

• PINNs中的梯度病态与梯度流的刚性相关，导致损失项之间的梯度不平衡。
• 当边界/初始条件梯度消失时，PDE残差项可能主导训练，导致错误预测。
• 一种自适应学习率退火算法，在线调节损失项权重，稳定训练并平衡贡献。
• 一种新颖的网络架构相比标准全连接网络降低了梯度流动刚性。
• 所提方法在多种计算物理问题上使PINNs的预测准确性提升稳定地达到50–100x。