[论文解读] Characterizing possible failure modes in physics-informed neural networks

Aditi S. Krishnapriyan, Amir Gholami+3 more|arXiv (Cornell University)|Sep 2, 2021

Model Reduction and Neural Networks被引用 351

一句话总结

论文表明，vanilla physics-informed neural networks (PINNs) 在对流、反应、扩散问题中由于软 PDE 约束带来的优化困难，可能无法捕捉非平凡物理，并提出 curriculum regularization 和 seq2seq learning 以显著提升性能。

ABSTRACT

Recent work in scientific machine learning has developed so-called physics-informed neural network (PINN) models. The typical approach is to incorporate physical domain knowledge as soft constraints on an empirical loss function and use existing machine learning methodologies to train the model. We demonstrate that, while existing PINN methodologies can learn good models for relatively trivial problems, they can easily fail to learn relevant physical phenomena for even slightly more complex problems. In particular, we analyze several distinct situations of widespread physical interest, including learning differential equations with convection, reaction, and diffusion operators. We provide evidence that the soft regularization in PINNs, which involves PDE-based differential operators, can introduce a number of subtle problems, including making the problem more ill-conditioned. Importantly, we show that these possible failure modes are not due to the lack of expressivity in the NN architecture, but that the PINN's setup makes the loss landscape very hard to optimize. We then describe two promising solutions to address these failure modes. The first approach is to use curriculum regularization, where the PINN's loss term starts from a simple PDE regularization, and becomes progressively more complex as the NN gets trained. The second approach is to pose the problem as a sequence-to-sequence learning task, rather than learning to predict the entire space-time at once. Extensive testing shows that we can achieve up to 1-2 orders of magnitude lower error with these methods as compared to regular PINN training.

研究动机与目标

• 评估 PINNs 在具有非平凡 PDE 系数的对流、反应和反应-扩散问题上的表现。
• 识别软 PDE 约束如何影响 PINNs 的优化与条件数。
• 确定 PINN 表达能力是否是瓶颈，还是优化地形导致失败。
• 提出并评估缓解策略以改进 PINN 的训练与准确性。
• 开源实现以帮助研究者的可重复性与拓展性。

提出的方法

• 将带有软 PDE 残差的 PINN 训练形式化为损失项，与数据损失和边界条件并列。
• 通过扰动 Hessian 的特征向量来分析损失景观，以观察优化难度。
• 评估两种缓解策略：逐步增强 PDE 约束强度的课程正则化，以及对较短时间段进行解预测的 seq2seq 学习。
• 在对流、反应和反应-扩散设置下比较 vanilla PINN、curriculum PINN 和 seq2seq PINN。
• 使用 L-BFGS 优化与一个4层、每层 50 个神经元、tanh 激活函数以及在周期域上的 collocation 点。

实验结果

研究问题

• RQ1 原始 PINN 训练是否能准确学习具有非平凡系数的对流、反应和反应-扩散 PDE 的解？
• RQ2 软 PDE 正则化如何影响 PINNs 的损失景观与优化？
• RQ3 观察到的失败是网络容量不足还是优化挑战造成的？
• RQ4 课程正则化与 seq2seq 学习是否提升 PINN 的性能与稳定性？
• RQ5 这些方法是否比标准 PINNs 更能捕捉尖锐或扩散特征？

主要发现

• PINNs 仅在极小的 PDE 系数下能得到较好解，对于中等难度的情形则失败。
• 增加软 PDE 正则化会使损失景观变得更复杂、越难优化，尤其是在较高系数时。
• 网络容量不是主要瓶颈；带有 PDE 基于约束的优化是核心问题。
• 课程正则化显著改善误差和损失景观的平滑度，有时达近两个数量级的提升。
• seq2seq 学习（时间推进）相较于一次性预测整个时空，在反应和反应-扩散问题中将误差降低多达近两个数量级。
• 开源实现提供可重复性与扩展机会。