[Paper Review] When and why PINNs fail to train: A neural tangent kernel perspective

Sifan Wang, Xinling Yu+1 more|arXiv (Cornell University)|Jul 28, 2020

Model Reduction and Neural Networks46 references120 citations

TL;DR

The paper analyzes PINN training dynamics through the neural tangent kernel (NTK) lens, proving convergence to a deterministic kernel in the infinite-width limit and identifying spectral-bias–related training pathologies, then offering an adaptive NTK-based training strategy.

ABSTRACT

Physics-informed neural networks (PINNs) have lately received great attention thanks to their flexibility in tackling a wide range of forward and inverse problems involving partial differential equations. However, despite their noticeable empirical success, little is known about how such constrained neural networks behave during their training via gradient descent. More importantly, even less is known about why such models sometimes fail to train at all. In this work, we aim to investigate these questions through the lens of the Neural Tangent Kernel (NTK); a kernel that captures the behavior of fully-connected neural networks in the infinite width limit during training via gradient descent. Specifically, we derive the NTK of PINNs and prove that, under appropriate conditions, it converges to a deterministic kernel that stays constant during training in the infinite-width limit. This allows us to analyze the training dynamics of PINNs through the lens of their limiting NTK and find a remarkable discrepancy in the convergence rate of the different loss components contributing to the total training error. To address this fundamental pathology, we propose a novel gradient descent algorithm that utilizes the eigenvalues of the NTK to adaptively calibrate the convergence rate of the total training error. Finally, we perform a series of numerical experiments to verify the correctness of our theory and the practical effectiveness of the proposed algorithms. The data and code accompanying this manuscript are publicly available at \url{https://github.com/PredictiveIntelligenceLab/PINNsNTK}.

Motivation & Objective

• Investigate training dynamics of fully-connected PINNs under gradient descent using NTK theory.
• Derive the PINN NTK and show its convergence to a deterministic kernel at infinite width.
• Analyze how NTK spectrum governs convergence rates of loss components in PINNs.
• Identify spectral bias and the discrepancy between loss terms as a fundamental pathology.
• Propose an adaptive training strategy that uses NTK eigenvalues to balance convergence across loss components.

Proposed method

• Define the PINN loss as L = L_b + L_r with data and PDE residual terms.
• Derive the coupled gradient-flow dynamics and the NTK matrix K(t) that governs evolution.
• Prove that in the infinite-width limit the PINN outputs and PDE residual converge to Gaussian processes.
• Show that the PINN NTK converges to a deterministic kernel K* at initialization and remains constant during training for infinitesimal learning rate.
• Analyze the eigenstructure of K* to explain spectral bias and differential convergence rates of loss components.
• Propose an adaptive weighting scheme (λ_b, λ_r) guided by NTK spectrum to improve trainability.

Experimental results

Research questions

• RQ1 Under what conditions does the PINN NTK converge to a deterministic kernel in the infinite-width limit?
• RQ2 Does the PINN NTK remain constant during training, and what are the implications for training dynamics?
• RQ3 How does the spectrum of the PINN NTK affect the convergence rates of different loss components (boundary vs residual)?
• RQ4 Can adaptive weighting based on NTK eigenvalues mitigate spectral-bias and improve trainability?
• RQ5 What practical algorithm can be derived from NTK insights to enhance PINN training stability and accuracy?

Key findings

• PINNs converge to Gaussian processes at infinite width for linear PDEs.
• The PINN NTK converges to a deterministic kernel and stays constant during training with infinitesimally small learning rate.
• The total training error's convergence rate is governed by the spectrum of the NTK, revealing a discrepancy between loss components.
• PINNs exhibit spectral bias, with high-frequency components learned slowly due to rapid eigenvalue decay in the NTK.
• The authors propose an adaptive gradient-descent algorithm leveraging NTK eigenvalues to balance convergence across loss terms.
• Numerical experiments validate the theory and demonstrate improved trainability and accuracy with the proposed method.