[Paper Review] Galerkin Neural Networks: A Framework for Approximating Variational Equations with Error Control

Mark Ainsworth, Justin Dong|arXiv (Cornell University)|May 28, 2021

Model Reduction and Neural Networks23 references4 citations

TL;DR

This paper introduces Galerkin Neural Networks, a novel framework that iteratively constructs finite-dimensional subspaces using neural network realizations to solve variational equations via Galerkin approximation. By adaptively training neural networks to represent error indicators and using them as basis functions, the method achieves error-controlled convergence with minimal modification across diverse PDEs, including singular and high-order problems.

ABSTRACT

We present a new approach to using neural networks to approximate the solutions of variational equations, based on the adaptive construction of a sequence of finite-dimensional subspaces whose basis functions are realizations of a sequence of neural networks. The finite-dimensional subspaces are then used to define a standard Galerkin approximation of the variational equation. This approach enjoys a number of advantages, including: the sequential nature of the algorithm offers a systematic approach to enhancing the accuracy of a given approximation; the sequential enhancements provide a useful indicator for the error that can be used as a criterion for terminating the sequential updates; the basic approach is largely oblivious to the nature of the partial differential equation under consideration; and, some basic theoretical results are presented regarding the convergence (or otherwise) of the method which are used to formulate basic guidelines for applying the method.

Motivation & Objective

• To address the stagnation of relative error in neural network approximations of PDEs, typically plateauing around 0.1–1.0%, despite increasing network size.
• To develop a systematic, adaptive framework for improving neural network approximations without retraining larger networks from scratch.
• To provide a theoretically grounded, PDE-agnostic method that maintains error control through a posteriori estimates derived from neural network corrections.
• To enable seamless application across different PDE types, including second- and fourth-order equations, with varying spatial dimensions and singular data.

Proposed method

• The method constructs a sequence of finite-dimensional subspaces by training neural networks to approximate the dual representation of the weak residual, forming basis functions for Galerkin approximation.
• Each new basis function is derived by solving a variational problem that maximizes the residual in the energy norm, ensuring it captures the dominant error component.
• The error is estimated using a posteriori error indicators η(u_{i−1}, φ_i^{NN}), which are shown to be equivalent to the true energy norm error |||u − u_{i−1}|||.
• Neural networks are trained iteratively with decreasing learning rates and increasing width (n_i = 20 × 2^{i−1}), using box initialization and fixed quadrature rules for integration.
• The Galerkin solution is updated as u_i = u_{i−1} + α_i φ_i^{NN}, where α_i is computed via Galerkin projection to minimize the residual in the current subspace.
• The approach is agnostic to PDE type, order, or spatial dimension, relying only on a stable variational formulation and quadrature rules for integration.

Experimental results

Research questions

• RQ1 Can neural networks be systematically used to enrich finite-dimensional subspaces in a Galerkin method to improve approximation accuracy without retraining large networks?
• RQ2 How can a posteriori error estimates be constructed from neural network realizations to guide adaptive refinement and termination?
• RQ3 To what extent can this framework be applied uniformly across different PDEs, including high-order and singular problems, without architectural or algorithmic changes?
• RQ4 Does the error indicator η(u_{i−1}, φ_i^{NN}) reliably reflect the true energy norm error |||u − u_{i−1}|||, enabling effective convergence monitoring?
• RQ5 Can the method achieve sub-1% relative error in challenging PDEs such as biharmonic equations with point loads, where standard methods often stagnate?

Key findings

• The method achieves robust convergence across both second-order and fourth-order PDEs, including the biharmonic equation with a point load, where the solution exhibits logarithmic singularities.
• The estimated error η(u_{i−1}, φ_i^{NN}) closely tracks the true energy norm error |||u − u_{i−1}|||, validating its use as a reliable a posteriori error indicator.
• For the biharmonic problem with ε₁ = ε₂ = 10⁻⁵, the method achieved a final tolerance of tol = 5×10⁻³ in both L² and energy norms after 5 refinement iterations.
• The algorithm successfully handles singular data and boundary conditions without numerical instability, as demonstrated in the 2D circular plate with point load.
• The use of increasing network width (n_i = 20×2^{i−1}) and decreasing learning rate (α_i = 10⁻² / 1.1^{i−1}) enabled stable and accurate refinement across iterations.
• The framework is PDE-agnostic: no structural changes are needed when switching between different PDE types or spatial dimensions, unlike finite element methods.