[论文解读] Is it time to swish? Comparing activation functions in solving the Helmholtz equation using physics-informed neural networks

Ali Al-Safwan, Chao Song+1 more|arXiv (Cornell University)|Oct 11, 2021

Model Reduction and Neural Networks参考文献 7被引用 6

一句话总结

本文研究了不同激活函数对物理信息神经网络（PINNs）在地震波场建模中求解Helmholtz方程性能的影响。结果表明，与常用的tanh、atan和ELU等激活函数相比，swish激活函数显著提升了收敛速度和解的精度，尤其在具有陡峭速度非均质性的模型中表现更优。

ABSTRACT

Solving the wave equation numerically constitutes the majority of the computational cost for applications like seismic imaging and full waveform inversion. An alternative approach is to solve the frequency domain Helmholtz equation, since it offers a reduction in dimensionality as it can be solved per frequency. However, computational challenges with the classical Helmholtz solvers such as the need to invert a large stiffness matrix can make these approaches computationally infeasible for large 3D models or for modeling high frequencies. Moreover, these methods do not have a mechanism to transfer information gained from solving one problem to the next. This becomes a bottleneck for applications like full waveform inversion where repeated modeling is necessary. Therefore, recently a new approach based on the emerging paradigm of physics informed neural networks (PINNs) has been proposed to solve the Helmholtz equation. The method has shown promise in addressing several challenging associated with the conventional algorithms, including flexibility to model additional physics and the use of transfer learning to speed up computations. However, the approach still needs further developments to be fully practicable. Foremost amongst the challenges is the slow convergence speed and reduced accuracy, especially in presence of sharp heterogeneities in the velocity model. Therefore, with an eye on exploring how improved convergence can be obtained for the PINN Helmholtz solvers, we study different activation functions routinely used in the PINN literature, in addition to the swish activation function - a variant of ReLU that has shown improved performance on a number of data science problems. Through a comparative study, we find that swish yields superior performance compared to the other activation functions routinely used in the PINN literature.

研究动机与目标

• 为解决在存在陡峭速度非均质性时，基于PINN的Helmholtz求解器收敛速度慢且精度降低的问题。
• 评估不同激活函数对PINNs在波场建模中训练动态和解质量的影响。
• 确定已知在深度学习中表现更优的swish激活函数是否能提升PINN求解Helmholtz方程的性能。
• 为选择最优激活函数提供实证依据，以加速物理信息深度学习在波传播问题中的收敛并提高精度。

提出的方法

• 本研究采用物理信息神经网络（PINN）求解二维散射速度模型中Helmholtz方程的Lippmann-Schwinger形式。
• 通过损失函数强制满足Helmholtz方程的残差，使用自动微分计算空间导数。
• 采用八层全连接隐藏层，每层20个神经元，使用Adam优化器进行15,000个周期的训练，批量大小为256。
• 比较了五种激活函数：tanh、atan、ELU和swish（一种平滑的非单调ReLU变体），所有超参数保持一致以确保公平比较。
• 使用SciANN包（一个高级Keras/TensorFlow封装）实现PINN并计算损失函数。
• 通过损失历史曲线评估收敛性，使用L2和L∞范数量化波场实部与虚部的解精度。

实验结果

研究问题

• RQ1 与tanh、atan和ELU等标准激活函数相比，swish激活函数是否能提升基于PINN的Helmholtz求解器的收敛速度和精度？
• RQ2 在具有陡峭速度对比的Helmholtz方程求解中，不同激活函数如何影响PINN的训练动态？
• RQ3 swish的改进梯度流动特性是否能缓解在高频波场问题训练中PINN的梯度消失问题？
• RQ4 激活函数的选择在多大程度上影响了PINN在全波形反演等应用中重复波场建模的可行性？

主要发现

• swish激活函数实现了最快的收敛速率，在前2000个周期内损失下降最快，优于tanh、atan和ELU。
• swish在波场实部和虚部的L2误差均最低：实部为3.65×10⁻⁵，虚部为4.7×10⁻⁵，优于所有其他激活函数。
• swish还产生了最小的L∞误差：实部为9.86×10⁻³，虚部为1.58×10⁻²，表明最大误差控制更优。
• 如图3所示，采用swish激活函数的PINN在空间分布上表现出更低的波场误差幅值。
• 损失曲线表明，swish能更快达到更低的平台，从而可更早切换至二阶优化器（如L-BFGS），进一步加速收敛。
• 在标准激活函数中，ELU表现最差，L2和L∞误差最高；而tanh和atan性能相似但不如swish。