[论文解读] Learning in PINNs: Phase transition, total diffusion, and generalization

Sokratis Anagnostopoulos, Juan Diego Toscano+2 more|arXiv (Cornell University)|Mar 27, 2024

Ferroelectric and Negative Capacitance Devices被引用 10

一句话总结

本文通过梯度信噪比分析 PINN 的学习动态，识别出第三阶段，称为总扩散，并提出基于残差的再加权以改善泛化。

ABSTRACT

We investigate the learning dynamics of fully-connected neural networks through the lens of gradient signal-to-noise ratio (SNR), examining the behavior of first-order optimizers like Adam in non-convex objectives. By interpreting the drift/diffusion phases in the information bottleneck theory, focusing on gradient homogeneity, we identify a third phase termed ``total diffusion", characterized by equilibrium in the learning rates and homogeneous gradients. This phase is marked by an abrupt SNR increase, uniform residuals across the sample space and the most rapid training convergence. We propose a residual-based re-weighting scheme to accelerate this diffusion in quadratic loss functions, enhancing generalization. We also explore the information compression phenomenon, pinpointing a significant saturation-induced compression of activations at the total diffusion phase, with deeper layers experiencing negligible information loss. Supported by experimental data on physics-informed neural networks (PINNs), which underscore the importance of gradient homogeneity due to their PDE-based sample inter-dependence, our findings suggest that recognizing phase transitions could refine ML optimization strategies for improved generalization.

研究动机与目标

• 使用梯度信噪比（SNR）理解 PINN 的梯度基学习动态。
• 在训练过程中识别并表征阶段转换，包括提出的总扩散阶段。
• 提出基于残差的再加权方案以促进梯度/均匀性并改善泛化。
• 将 SNR 行为与信息压缩在网络激活中的关系联系起来。

提出的方法

• 对 PINN 的全批 gradient descent 和 Adam 优化器进行模型分析。
• 在训练过程中定义并测量梯度SNR 与残差同质性。
• 引入基于残差的注意力（RBA）再加权以促进样本间残差的均匀性。
• 分析信息瓶颈概念以将 SNR、扩散和激活压缩联系起来。
• 在 PINN 基准测试上进行实证测试（Allen-Cahn、Helmholtz、Burgers、lid-driven cavity）。
• 比较 vanilla 与 RBA 变体以评估扩散阶段的实现和泛化。

Figure 1: Phase transition in PINNs: The test error between the prediction and the exact solution converges faster after total diffusion (dashed lines), which occurs with an abrupt phase transition defined by homogeneous residuals. Although the convergence starts during the onset of the diffusion ph

实验结果

研究问题

• RQ1 在用 Adam 训练的 PINN 中，梯度动态的阶段转换（拟合、扩散、总扩 diffusion）会发生吗？
• RQ2 梯度同质性和残差扩散在实现最优收敛和泛化中的作用是什么？
• RQ3 基于残差的再加权是否能加速扩散并改善 PINN 的泛化？
• RQ4 SNR 动态如何与 PINN 的信息压缩和激活二值化相关？

主要发现

• 在经过较长训练并出现突发的 SNR 增加与梯度同质性后，出现第三阶段，总扩散。
• 基于残差的注意力（RBA）通过促进残差同质性来加速扩散并改善泛化。
• 跨样本的梯度同质性与更好收敛和更均匀的训练动态相关。
• SNR 行为与信息压缩和激活饱和相关，且更深层在总扩散阶段的信息损失较小。
• PINN 在取样点之间存在梯度相互依赖，影响优化稳定性和泛化。
• 在实验中，RBA 模型更快达到总扩散，在大多数基准上表现出更好的泛化。

Figure 2: Gradient-based optimization regimes: Indicative SNR training curve at each full-batch iteration. For $\text{SNR}\gg 1$ , the deterministic term dominates, while for $\text{SNR}\ll 1$ , each step becomes more stochastic. The first two stages of learning are defined as “fitting” ( $\text{SNR