[Paper Review] Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory

Tao Luo, Haizhao Yang|arXiv (Cornell University)|Jun 28, 2020

Model Reduction and Neural NetworksPhysics and Astronomy49 references50 citations

TL;DR

The paper proves that gradient descent on over-parameterized two-layer neural networks can find a global minimizer for solving second-order linear PDEs via least-squares formulation, and it analyzes generalization under Barron-type spaces.

ABSTRACT

The problem of solving partial differential equations (PDEs) can be formulated into a least-squares minimization problem, where neural networks are used to parametrize PDE solutions. A global minimizer corresponds to a neural network that solves the given PDE. In this paper, we show that the gradient descent method can identify a global minimizer of the least-squares optimization for solving second-order linear PDEs with two-layer neural networks under the assumption of over-parametrization. We also analyze the generalization error of the least-squares optimization for second-order linear PDEs and two-layer neural networks, when the right-hand-side function of the PDE is in a Barron-type space and the least-squares optimization is regularized with a Barron-type norm, without the over-parametrization assumption.

Motivation & Objective

• Formulate PDE solving as a least-squares problem using neural networks to parametrize PDE solutions.
• Show convergence of gradient descent to a global minimizer under over-parameterization for second-order linear PDEs.
• Develop generalization bounds for PDE solvers using Barron-type spaces and path-norm regularization.
• Extend neural tangent kernel-inspired analysis to PDE solver settings with boundary conditions.
• Provide a framework for comparing empirical and population losses in PDE contexts.

Proposed method

• Use a two-layer neural network to parametrize PDE solutions and formulate a population loss combining interior and boundary operators.
• Transform boundary-condition-heavy problems into a form where the network inherently satisfies certain boundary conditions via specially designed auxiliary functions.
• Prove linear convergence of gradient descent to a global minimizer of the empirical loss under over-parameterization and bounded operator assumptions.
• Introduce Barron-type function spaces and a path norm to quantify complexity and enable generalization bounds without truncation tricks.
• Derive a posteriori and a priori generalization bounds using Rademacher complexity for the PDE solver setting.
• Relate empirical minimizers to population minimizers through Barron-type norms and regularization.

Experimental results

Research questions

• RQ1 Can gradient descent identify a global minimizer for least-squares PDE solvers with over-parameterized two-layer networks?
• RQ2 How does the empirical minimizer compare to the population minimizer when the PDE right-hand side lies in a Barron-type space?
• RQ3 What role do boundary conditions play when designing networks that automatically satisfy them in PDE solvers?
• RQ4 What are the generalization guarantees for PDE solvers using Barron-type norms and path-norm regularization?
• RQ5 How can Rademacher complexity be employed to bound generalization error in PDE neural solvers?

Key findings

• Under over-parameterization, gradient descent achieves linear convergence to a global minimizer of the empirical PDE loss for second-order linear PDEs.
• The a posteriori generalization gap is bounded by a term that scales with the path norm and inversely with the square root of sample size.
• If the RHS lies in a Barron-type space and a Barron-type regularization is used, the a priori generalization bound scales with the Barron norm of the target function.
• Special boundary-condition-design networks eliminate the need to balance PDE and boundary terms via hyperparameters in the loss.
• The analysis extends neural tangent kernel ideas and Barron-space generalization theory to PDE solvers with variable coefficients and second-order operators.