[论文解读] Physical Activation Functions (PAFs): An Approach for More Efficient Induction of Physics into Physics-Informed Neural Networks (PINNs)

Jassem Abbasi, Pål Østebø Andersen|arXiv (Cornell University)|May 29, 2022

Machine Learning in Materials Science被引用 9

一句话总结

本文提出物理激活函数（PAFs），这是一种新颖的方法，通过从解析解或边界条件推导其数学形式，将物理定律直接嵌入物理信息神经网络（PINNs）的激活函数中。通过用受物理启发的函数替代标准激活函数（如ReLU），PAFs显著提升了PINN的训练效率，模型规模最多减少75%，损失值降低1至2个数量级，从而增强了对分布外数据的泛化能力。

ABSTRACT

In recent years, the gap between Deep Learning (DL) methods and analytical or numerical approaches in scientific computing is tried to be filled by the evolution of Physics-Informed Neural Networks (PINNs). However, still, there are many complications in the training of PINNs and optimal interleaving of physical models. Here, we introduced the concept of Physical Activation Functions (PAFs). This concept offers that instead of using general activation functions (AFs) such as ReLU, tanh, and sigmoid for all the neurons, one can use generic AFs that their mathematical expression is inherited from the physical laws of the investigating phenomena. The formula of PAFs may be inspired by the terms in the analytical solution of the problem. We showed that the PAFs can be inspired by any mathematical formula related to the investigating phenomena such as the initial or boundary conditions of the PDE system. We validated the advantages of PAFs for several PDEs including the harmonic oscillations, Burgers, Advection-Convection equation, and the heterogeneous diffusion equations. The main advantage of PAFs was in the more efficient constraining and interleaving of PINNs with the investigating physical phenomena and their underlying mathematical models. This added constraint significantly improved the predictions of PINNs for the testing data that was out-of-training distribution. Furthermore, the application of PAFs reduced the size of the PINNs up to 75% in different cases. Also, the value of loss terms was reduced by 1 to 2 orders of magnitude in some cases which is noteworthy for upgrading the training of the PINNs. The iterations required for finding the optimum values were also significantly reduced. It is concluded that using the PAFs helps in generating PINNs with less complexity and much more validity for longer ranges of prediction.

研究动机与目标

• 解决由于物理定律与神经网络架构之间交错不良而导致的物理信息神经网络（PINNs）训练低效与不稳定的挑战。
• 在保持或提升预测性能的同时，降低PINN架构的复杂度与规模。
• 探索一种新范式：激活函数不再通用，而是基于物理原理或底层偏微分方程（PDEs）的解析解推导而来。
• 证明PAFs在多个基准PDE上可实现PINN训练的更快收敛与更低损失值。

提出的方法

• 通过推导待求解PDE的已知解析解、初始条件或边界条件，构建激活函数的函数形式。
• 不使用ReLU或tanh等标准激活函数，而是采用具有物理意义的数学函数，以反映系统的真实物理行为。
• 将PAFs集成到神经网络架构中，应用于隐藏神经元，从而将物理约束直接嵌入网络的归纳偏置。
• 在多个PDE上验证该方法，包括谐振荡、Burgers方程、对流-扩散方程以及非均匀扩散方程。
• 采用标准PINN损失最小化进行训练，但用PAFs替代标准激活函数，从一开始就强制实现物理一致性。
• 通过损失降低程度、模型大小、收敛速度以及对分布外测试数据的泛化能力来评估性能。

实验结果

研究问题

• RQ1 能否通过将物理定律直接嵌入激活函数来提升PINN的训练效率与精度？
• RQ2 在保持或提升预测性能的前提下，PAFs在多大程度上可减少PINN架构的规模与复杂度？
• RQ3 与标准激活函数相比，PAFs如何影响PINN的收敛速度与损失曲面特性？
• RQ4 PAFs能否增强PINN对分布外数据的泛化能力，特别是在训练集中代表性不足的区域？
• RQ5 与基线模型相比，PAFs对PINN训练中损失项的大小有何影响？

主要发现

• 在多个测试案例中，PAFs使PINN模型规模最多减少75%，显著降低了模型复杂度。
• 使用PAFs后，PINN训练中的损失值降低了1至2个数量级，表明优化效果更好且约束满足更充分。
• 收敛速度显著加快，达到最优训练性能所需的迭代次数大幅减少。
• 采用PAFs的PINNs在分布外测试数据上表现出更优的泛化能力，而标准PINNs则在这些情况下失效。
• 由PAFs引入的物理归纳偏置，使模型在更长预测范围内产生更合理且物理一致的预测结果。
• 该方法在多种PDE上成功验证，包括谐振荡、Burgers方程、对流-扩散方程以及非均匀扩散方程。