[Paper Review] Understanding and Mitigating Extrapolation Failures in Physics-Informed Neural Networks

Lukas Fesser, Luca D’Amico-Wong+1 more|arXiv (Cornell University)|Jun 15, 2023

Model Reduction and Neural Networks4 citations

TL;DR

This paper identifies spectral shifts in the Fourier spectrum of solution functions—not high-frequency content—as the primary cause of extrapolation failures in Physics-Informed Neural Networks (PINNs). It introduces the Weighted Wasserstein-Fourier (WWF) distance to quantify these shifts and demonstrates that transfer learning based on similar PDEs can reduce extrapolation errors by up to 82% when spectral shifts are present.

ABSTRACT

Physics-informed Neural Networks (PINNs) have recently gained popularity due to their effective approximation of partial differential equations (PDEs) using deep neural networks (DNNs). However, their out of domain behavior is not well understood, with previous work speculating that the presence of high frequency components in the solution function might be to blame for poor extrapolation performance. In this paper, we study the extrapolation behavior of PINNs on a representative set of PDEs of different types, including high-dimensional PDEs. We find that failure to extrapolate is not caused by high frequencies in the solution function, but rather by shifts in the support of the Fourier spectrum over time. We term these spectral shifts and quantify them by introducing a Weighted Wasserstein-Fourier distance (WWF). We show that the WWF can be used to predict PINN extrapolation performance, and that in the absence of significant spectral shifts, PINN predictions stay close to the true solution even in extrapolation. Finally, we propose a transfer learning-based strategy to mitigate the effects of larger spectral shifts, which decreases extrapolation errors by up to 82%.

Motivation & Objective

• To investigate the root cause of extrapolation failures in PINNs beyond the commonly assumed role of high-frequency components.
• To characterize PDEs that enable or hinder PINN extrapolation using Fourier spectral analysis.
• To develop a quantifiable metric—Weighted Wasserstein-Fourier distance (WWF)—to predict extrapolation performance.
• To evaluate whether transfer learning can mitigate extrapolation errors caused by spectral shifts in PDE solutions.
• To demonstrate the generalization of findings across low- and high-dimensional PDEs.

Proposed method

• The authors analyze the Fourier spectra of solution functions across various PDEs to detect shifts in spectral support over time.
• They introduce the Weighted Wasserstein-Fourier (WWF) distance to quantify spectral shifts between interpolation and extrapolation domains.
• PINNs are trained on time-dependent PDEs with known initial and boundary conditions, and their extrapolation performance is evaluated beyond the training time horizon.
• Transfer learning is applied by pre-training on a family of similar PDEs (e.g., varying parameters like diffusion coefficient or Reynolds number) before fine-tuning on the target PDE.
• Experiments compare baseline PINNs, half-domain transfer learning, and full-domain transfer learning to assess improvements in extrapolation accuracy.
• The WWF distance is computed between the Fourier spectra of the solution in the training and extrapolation intervals to correlate with observed extrapolation errors.

Experimental results

Research questions

• RQ1 What causes extrapolation failures in PINNs beyond the presence of high-frequency components in the solution?
• RQ2 Can spectral shifts in the Fourier spectrum of PDE solutions predict PINN extrapolation performance?
• RQ3 Does transfer learning improve extrapolation when spectral shifts are present in the solution dynamics?
• RQ4 How does the presence or absence of spectral shifts affect the performance of transfer learning in PINNs?
• RQ5 Can the proposed WWF distance serve as a reliable predictor of extrapolation error in PINNs?

Key findings

• Extrapolation failures in PINNs are not caused by high-frequency content in the solution function, contradicting prior speculation.
• Spectral shifts—changes in the support of the Fourier spectrum over time—are the primary driver of extrapolation errors in PINNs.
• The Weighted Wasserstein-Fourier (WWF) distance successfully quantifies spectral shifts and predicts PINN extrapolation performance with high accuracy.
• When spectral shifts are absent, transfer learning provides no improvement in extrapolation, confirming that spectral shifts are the key target for mitigation.
• Transfer learning based on similar PDEs reduces extrapolation errors by up to 82% when significant spectral shifts are present.
• The benefits of transfer learning are specific to extrapolation; in interpolation, transfer learning can degrade performance due to stronger inductive biases.