[Paper Review] An analysis of the derivative-free loss method for solving PDEs

Jihun Han, Yoonsang Lee|arXiv (Cornell University)|Sep 28, 2023

Model Reduction and Neural NetworksPhysics and Astronomy3 citations

TL;DR

This paper analyzes the derivative-free loss method (DFLM) for solving elliptic PDEs using neural networks via the Feynman-Kac formulation. It establishes that training loss bias is proportional to the time interval Δt and inversely proportional to walker count Ns, and shows Δt must be sufficiently large for effective training, enabling minimal Ns for computational efficiency when Δt is optimally bounded.

ABSTRACT

This study analyzes the derivative-free loss method to solve a certain class of elliptic PDEs and fluid problems using neural networks. The approach leverages the Feynman-Kac formulation, incorporating stochastic walkers and their averaged values. We investigate how the time interval associated with the Feynman-Kac representation and the walker size influence computational efficiency, trainability, and sampling errors. Our analysis shows that the training loss bias scales proportionally with the time interval and the spatial gradient of the neural network, while being inversely proportional to the walker size. Moreover, we demonstrate that the time interval must be sufficiently long to enable effective training. These results indicate that the walker size can be chosen as small as possible, provided it satisfies the optimal lower bound determined by the time interval. Finally, we present numerical experiments that support our theoretical findings.

Motivation & Objective

• To analyze the training loss bias in the derivative-free loss method (DFLM) for solving elliptic PDEs using neural networks.
• To investigate the influence of the time interval Δt and walker count Ns on computational efficiency, trainability, and sampling error.
• To determine the minimal required Δt for effective training and optimal Ns selection.
• To provide theoretical bounds and numerical validation for the interplay between Δt, Ns, and solution accuracy.

Proposed method

• Uses the Feynman-Kac formulation to represent the PDE solution as an expected value of stochastic walkers.
• Employs Ns stochastic walkers per collocation point to estimate the expected value in the Feynman-Kac formula.
• Applies bootstrapping-like iterative refinement, where the neural network's target is updated using its current predictions.
• Analyzes the bias in the empirical training loss as a function of Δt and Ns using theoretical bounds.
• Uses numerical experiments with varying Δt and Ns to validate theoretical findings on test error and convergence.
• Considers both smooth and highly oscillatory solutions to assess sensitivity to local solution variations.

Experimental results

Research questions

• RQ1 How does the training loss bias in DFLM scale with the time interval Δt and the number of walkers Ns?
• RQ2 What is the minimum required Δt for the neural network to be effectively trainable in DFLM?
• RQ3 Can Ns be minimized without sacrificing accuracy, given an optimal Δt?
• RQ4 How do local solution variations, such as high gradients or oscillations, affect the required Δt?
• RQ5 What is the relationship between the optimal Δt and the wavenumber or spatial frequency of the solution?

Key findings

• The training loss bias is bounded by a term proportional to Δt and inversely proportional to Ns, confirming theoretical scaling.
• When Ns = 1, the training loss bias is non-negligible, and the minimal test error occurs at Δt ≈ 5×10⁻³.
• For larger Ns (e.g., 400), the test error is minimized when Δt ≥ 5×10⁻³, indicating a lower bound on Δt.
• As Δt decreases, test error increases regardless of Ns, showing that Δt must be sufficiently large for training to be effective.
• The optimal Δt for a more oscillatory solution (m=3) is about ten times smaller than for a smoother solution (m=1), consistent with √Δt scaling of walker spread.
• The variance of walkers is proportional to Δt, and the optimal Δt scales inversely with the square of the solution’s wavenumber, explaining the observed factor-of-ten difference.