[Paper Review] Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks

Ghazal Farhani, Alexander Kazachek+1 more|arXiv (Cornell University)|Jun 29, 2022

Model Reduction and Neural Networks4 citations

TL;DR

This paper demonstrates that stochastic gradient descent with momentum (SGDM) significantly reduces spectral bias in physics-informed neural networks (PINNs), enabling convergence to accurate solutions even for high-frequency partial differential equation (PDE) problems. Using neural tangent kernel (NTK) theory, the authors show that SGDM improves learning dynamics by balancing low- and high-frequency feature convergence, outperforming vanilla SGD and enabling wider networks to achieve high-accuracy solutions.

ABSTRACT

Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs). However, they often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias. In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under stochastic gradient descent with momentum (SGDM). This demonstrates SGDM significantly reduces the effect of spectral bias. We have also examined why training a model via the Adam optimizer can accelerate the convergence while reducing the spectral bias. Moreover, our numerical experiments have confirmed that wide-enough networks using SGDM still converge to desirable solutions, even in the presence of high-frequency features. In fact, we show that the width of a network plays a critical role in convergence.

Motivation & Objective

• Address the persistent issue of spectral bias in PINNs, where models fail to learn high-frequency features in PDE solutions.
• Investigate why standard SGD struggles with high-frequency PDEs due to strong spectral bias in training dynamics.
• Explore how momentum in optimization (SGDM) and adaptive methods (Adam) mitigate spectral bias and accelerate convergence.
• Demonstrate that network width plays a critical role in reducing error for high-frequency problems, even in practical, finite-width networks.
• Provide theoretical and empirical evidence that SGDM enables convergence to accurate solutions for high-frequency PDEs by modifying gradient flow dynamics.

Proposed method

• Employ neural tangent kernel (NTK) theory to analyze the gradient flow dynamics of infinitely-wide PINNs under SGDM optimization.
• Model the training process as a continuous-time dynamical system using NTK-based analysis to study convergence behavior across frequency components.
• Compare the convergence rates of low- and high-frequency features under SGDM, showing that momentum slows down low-frequency learning while accelerating high-frequency components.
• Use numerical experiments with fixed-depth, finite-width networks to validate theoretical findings on the impact of width and optimizer choice.
• Implement PINNs with SGDM and Adam optimizers on benchmark PDEs (Poisson, advection, reaction-diffusion) with increasing frequency parameters.
• Quantify solution accuracy via relative error between predicted and analytical solutions across varying network widths and hyperparameters.

Experimental results

Research questions

• RQ1 How does stochastic gradient descent with momentum (SGDM) affect spectral bias in physics-informed neural networks (PINNs) for solving PDEs?
• RQ2 Why does the Adam optimizer accelerate convergence in PINNs compared to vanilla SGD, and how does it influence spectral bias?
• RQ3 To what extent does increasing the width of a PINN mitigate spectral bias and improve solution accuracy for high-frequency PDEs?
• RQ4 Can the theoretical convergence dynamics of infinitely-wide PINNs under SGDM be extended to practical, finite-width networks?
• RQ5 How do the training dynamics of low- and high-frequency components differ under SGDM, and what role does momentum play in balancing them?

Key findings

• SGDM significantly reduces spectral bias in PINNs by slowing down the convergence of low-frequency features and accelerating high-frequency ones, leading to more balanced learning across frequency components.
• For high-frequency PDEs, vanilla SGD fails to converge due to extremely slow learning rates required for high-frequency components, while SGDM enables convergence even in challenging cases.
• Numerical experiments confirm that wider networks (e.g., width 500) achieve relative errors on the order of 10⁻³ for high-frequency problems, whereas narrower networks (width 50) yield errors on the order of 10⁻¹.
• In reaction-diffusion and transport equations with high parameters (e.g., ν=23, ρ=25, β=100), SGDM and Adam achieved significantly lower errors than vanilla SGD after fewer epochs.
• The use of SGDM leads to faster convergence than Adam in some cases, though Adam still outperforms vanilla SGD due to adaptive learning rates.
• Theoretical analysis via NTK shows that SGDM induces under-damped and critically-damped oscillatory dynamics for low- and high-frequency components, respectively, reducing spectral bias compared to vanilla SGD.