[论文解读] Bias-Variance Trade-off in Physics-Informed Neural Networks with Randomized Smoothing for High-Dimensional PDEs

Zheyuan Hu, Zhouhao Yang+3 more|arXiv (Cornell University)|Nov 26, 2023

Model Reduction and Neural Networks被引用 9

一句话总结

本文分析了高维偏微分方程中的随机平滑 PINN（RS-PINN）的偏差，提出去偏差技术和一种混合有偏-无偏方法，并在若干高维偏微分方程上进行验证。

ABSTRACT

While physics-informed neural networks (PINNs) have been proven effective for low-dimensional partial differential equations (PDEs), the computational cost remains a hurdle in high-dimensional scenarios. This is particularly pronounced when computing high-order and high-dimensional derivatives in the physics-informed loss. Randomized Smoothing PINN (RS-PINN) introduces Gaussian noise for stochastic smoothing of the original neural net model, enabling Monte Carlo methods for derivative approximation, eliminating the need for costly auto-differentiation. Despite its computational efficiency in high dimensions, RS-PINN introduces biases in both loss and gradients, negatively impacting convergence, especially when coupled with stochastic gradient descent (SGD). We present a comprehensive analysis of biases in RS-PINN, attributing them to the nonlinearity of the Mean Squared Error (MSE) loss and the PDE nonlinearity. We propose tailored bias correction techniques based on the order of PDE nonlinearity. The unbiased RS-PINN allows for a detailed examination of its pros and cons compared to the biased version. Specifically, the biased version has a lower variance and runs faster than the unbiased version, but it is less accurate due to the bias. To optimize the bias-variance trade-off, we combine the two approaches in a hybrid method that balances the rapid convergence of the biased version with the high accuracy of the unbiased version. In addition, we present an enhanced implementation of RS-PINN. Extensive experiments on diverse high-dimensional PDEs, including Fokker-Planck, HJB, viscous Burgers', Allen-Cahn, and Sine-Gordon equations, illustrate the bias-variance trade-off and highlight the effectiveness of the hybrid RS-PINN. Empirical guidelines are provided for selecting biased, unbiased, or hybrid versions, depending on the dimensionality and nonlinearity of the specific PDE problem.

研究动机与目标

• 识别 RS-PINN 中源于 MSE 非线性和 PDE 非线性的偏差来源。
• 为线性和非线性 PDE 开发去偏差策略以纠正已识别的偏差。
• 提出一种混合有偏-无偏的 RS-PINN，以优化收敛速度与精度。
• 通过改进的、较低方差的导数估计，扩展 RS-PINN 的实现。
• 为根据问题特征在有偏、无偏或混合 RS-PINN 之间选择提供实证性指南。

提出的方法

• 将 RS-PINN 模型化为带高斯噪声的平滑神经网络，并对导数进行蒙特卡洛估计。
• 将偏差分解为两种来源：MSE 损失非线性和 PDE 非线性。
• 通过使用两个独立的高斯样本来形成蒙特卡洛估计的乘积（Lb^(1)），对 Lb 进行去偏差并证明其无偏性。
• 通过对非线性 PDE 的平方梯度项进行去偏差，使用多组高斯样本（Lr^(2)）实现去偏差；推广到具有 n 次非线性时的通用性。
• 引入一种混合方案，初期有偏以实现快速收敛，随后切换到无偏以提高精度。
• 提供对导数的分离 x- 与 t-噪声处理的实现改进，以降低方差。

Figure 2: Anisotropic FP PDE: $10^{4}$ D convergence curves with respect to the epoch (left) and time (right). The hybrid version converges well by applying the biased version first; then, the unbiased version is used for finetuning and getting an even more stable final convergence result. Solely ap

实验结果

研究问题

• RQ1 在解决高维 PDE 时，RS-PINN 的主要偏差来源是什么？
• RQ2 如何在线性与非线性 PDE 情况下对 RS-PINN 的损失及其梯度进行去偏差？
• RQ3 在不同维度和非线性阶数下，带偏-无偏混合 RS-PINN 相对于单独有偏或无偏方法的表现如何？
• RQ4 可以基于维数和 PDE 非线性给出哪些实用指南，用于选择 RS-PINN 的变体？

主要发现

• 有偏的 RS-PINN 具有较低方差和更快的每轮运行时间，但由于偏差可能导致精度下降。
• 无偏的 RS-PINN 在消除偏差的同时代价是方差更高、收敛速度较慢，尤其在高维情形。
• 来自 MSE 非线性造成的偏差可以通过独立重新采样（Lb^(1)）去偏；对 PDE 非线性，需要额外采样（Lg^(2)）以获得无偏梯度。
• 混合有偏-无偏的 RS-PINN 结合了初期的快速收敛（有偏）和最终的高精度（无偏）。
• 在高维 Fokker-Planck、HJB、粘性 Burgers、Allen-Cahn 和 Sine-Gordon 方程上的实验展示了偏差-方差权衡及混合方法的有效性。