[论文解读] When and why PINNs fail to train: A neural tangent kernel perspective

Sifan Wang, Xinling Yu+1 more|arXiv (Cornell University)|Jul 28, 2020

Model Reduction and Neural Networks参考文献 46被引用 120

一句话总结

论文通过神经切线核（NTK）视角分析 PINN 的训练动态，证明在无限宽度极限下收敛到一个确定性核，并识别与谱偏差相关的训练病理性问题，然后给出基于 NTK 的自适应训练策略。

ABSTRACT

Physics-informed neural networks (PINNs) have lately received great attention thanks to their flexibility in tackling a wide range of forward and inverse problems involving partial differential equations. However, despite their noticeable empirical success, little is known about how such constrained neural networks behave during their training via gradient descent. More importantly, even less is known about why such models sometimes fail to train at all. In this work, we aim to investigate these questions through the lens of the Neural Tangent Kernel (NTK); a kernel that captures the behavior of fully-connected neural networks in the infinite width limit during training via gradient descent. Specifically, we derive the NTK of PINNs and prove that, under appropriate conditions, it converges to a deterministic kernel that stays constant during training in the infinite-width limit. This allows us to analyze the training dynamics of PINNs through the lens of their limiting NTK and find a remarkable discrepancy in the convergence rate of the different loss components contributing to the total training error. To address this fundamental pathology, we propose a novel gradient descent algorithm that utilizes the eigenvalues of the NTK to adaptively calibrate the convergence rate of the total training error. Finally, we perform a series of numerical experiments to verify the correctness of our theory and the practical effectiveness of the proposed algorithms. The data and code accompanying this manuscript are publicly available at \url{https://github.com/PredictiveIntelligenceLab/PINNsNTK}.

研究动机与目标

• 使用NTK理论在梯度下降下研究全连接PINN的训练动态。
• 推导PINN NTK并证明在无限宽度时收敛到一个确定性核。
• 分析NTK谱如何支配PINN中损失分量的收敛速率。
• 指出谱偏差以及损失项之间的差异作为根本病理。
• 提出一个利用NTK特征值来平衡各损失分量收敛的自适应训练策略。

提出的方法

• 将PINN损失定义为 L = L_b + L_r，含数据和PDE残差项。
• 推导耦合的梯度流动力学以及控制演化的NTK矩阵K(t)。
• 证明在无限宽度极限下PINN输出和PDE残余收敛到高斯过程。
• 证明PINN NTK在初始化时收敛到确定性核K*，且在极小学习率下训练过程中保持恒定。
• 分析K*的特征结构以解释谱偏差和损失分量的差异收敛速率。
• 提出一个受NTK谱引导的自适应权重方案（λ_b, λ_r）以提升可训练性。

实验结果

研究问题

• RQ1 在哪些条件下，PINN NTK 在无限宽度极限下收敛到确定性核？
• RQ2 PINN NTK 在训练过程中保持不变吗，对训练动力学有什么影响？
• RQ3 PINN NTK 的谱如何影响不同损失分量（边界 vs 残差）的收敛速率？
• RQ4 基于NTK特征值的自适应加权能否缓解谱偏差并提升可训练性？
• RQ5 可以从NTK洞见推导出哪些实际算法以提升PINN训练的稳定性和准确性？

主要发现

• PINNs 收敛到高斯过程在无限宽度时对线性PDE。
• PINN NTK 收敛到确定性核并在学习率极小时训练中保持恒定。
• 总训练误差的收敛速率由NTK谱支配，揭示损失分量之间的差异。
• PINNs 呈现谱偏差，高频分量因NTK中特征值快速衰减而学习较慢。
• 作者提出一个利用NTK特征值实现跨损失项收敛平衡的自适应梯度下降算法。
• 数值实验验证理论，并展示所提方法在可训练性和精度上的改进。