[论文解读] Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism

Levi D. McClenny, Ulisses Braga-Neto|arXiv (Cornell University)|Sep 7, 2020

Model Reduction and Neural Networks参考文献 26被引用 42

一句话总结

本文提出自适应PINN（SA-PINN），它学习可训练的逐点权重以创建软注意力掩码，最大化对困难区域的强调，同时最小化整体损失，以在更少的迭代中更准确地求解刚性偏微分方程。

ABSTRACT

Physics-Informed Neural Networks (PINNs) have emerged recently as a promising application of deep neural networks to the numerical solution of nonlinear partial differential equations (PDEs). However, it has been recognized that adaptive procedures are needed to force the neural network to fit accurately the stubborn spots in the solution of "stiff" PDEs. In this paper, we propose a fundamentally new way to train PINNs adaptively, where the adaptation weights are fully trainable and applied to each training point individually, so the neural network learns autonomously which regions of the solution are difficult and is forced to focus on them. The self-adaptation weights specify a soft multiplicative soft attention mask, which is reminiscent of similar mechanisms used in computer vision. The basic idea behind these SA-PINNs is to make the weights increase as the corresponding losses increase, which is accomplished by training the network to simultaneously minimize the losses and maximize the weights. In addition, we show how to build a continuous map of self-adaptive weights using Gaussian Process regression, which allows the use of stochastic gradient descent in problems where conventional gradient descent is not enough to produce accurate solutions. Finally, we derive the Neural Tangent Kernel matrix for SA-PINNs and use it to obtain a heuristic understanding of the effect of the self-adaptive weights on the dynamics of training in the limiting case of infinitely-wide PINNs, which suggests that SA-PINNs work by producing a smooth equalization of the eigenvalues of the NTK matrix corresponding to the different loss terms. In numerical experiments with several linear and nonlinear benchmark problems, the SA-PINN outperformed other state-of-the-art PINN algorithm in L2 error, while using a smaller number of training epochs.

研究动机与目标

• 动机并解决基线PINN在刚性PDE上的收敛性和精度问题。
• 提出完全可训练的逐点自适应权重，在训练过程中强调困难区域。
• 发展连续的自适应掩码策略，并将其与偏微分方程约束优化理论联系起来。
• 提供一个实用的训练框架以及对SA-PINN训练动力学的理论见解。

提出的方法

• 为初始点、边界点和残差点定义带有逐点自适应权重的损失。
• 使用非负、可微的掩码 m(λ) 来按可训练的 λ 作为函数缩放每个点的损失。
• 通过对 w 求最小、对 λ 求最大以实现鞍点，实质上是一种罚函数方法。
• 推导 λ 的梯度，表明它们随相应的未掩蔽损失增大并描述单调权重增长。
• 将自适应权重映射到高斯过程，以在对随机梯度下降友好的训练中实现连续权重。
• 讨论神经切线核（NTK）对理解SA-PINN下训练动力学的影响。

实验结果

研究问题

• RQ1 逐点可训练权重是否能改善PINN在刚性PDE上的训练与收敛？
• RQ2 自适应权重如何影响损失分量的平衡和训练动力学？
• RQ3 与基线PINN和先前加权方案相比，SA-PINN对基准刚性PDE的影响如何？
• RQ4 基于GP的自适应权重映射能否实现对SA-PINN的有效SGD训练？

主要发现

• SA-PINNs 在 Allen-Cahn 方程上比基线显著降低 L2 误差（2.1% ± 1.21% 对比 基线的 96.15% ± 6.45%，以及非自适应加权的 49.61% ± 2.50%）。
• 在 Burgers 方程中，SA-PINNs 的 L2 误差为 4.803e-04 ± 1.01e-4，优于基线且用更少的训练轮次。
• 对于 Helmholtz 方程，SA-PINN 实现相对 L2 误差 3.2e-3 ± 2.2e-4，在较少迭代次数下接近先进方案的精度。
• SA-PINN 学习出可解释的权重图，强调困难区域（如 Burgers 的不连续性）和 Allen-Cahn 的早期时刻。
• NTK 分析表明 SA-PINN 能使各损失分量的特征值分布趋于均匀与平滑，帮助训练动力学。
• 作者提供开源实现，并在多个基准测试中显示出一致改进。