[Paper Review] Exponential ReLU Neural Network Approximation Rates for Point and Edge Singularities

Carlo Marcati, Joost A. A. Opschoor+2 more|arXiv (Cornell University)|Oct 23, 2020

Machine Learning in Materials Science4 citations

TL;DR

This paper establishes exponential convergence rates for ReLU neural network approximations of functions with point and edge singularities in 2D and 3D polytopal domains. It proves that deep ReLU networks achieve error decay of order $ C\exp(-bM^{1/(2d+1)}) $ in $ H^1 $-norm, where $ M $ is the number of network parameters, providing a theoretical foundation for deep learning in solving elliptic PDEs with analytic data and singular solutions.

ABSTRACT

We prove exponential expressivity with stable ReLU Neural Networks (ReLU NNs) in $H^1(Ω)$ for weighted analytic function classes in certain polytopal domains $Ω$, in space dimension $d=2,3$. Functions in these classes are locally analytic on open subdomains $D\subset Ω$, but may exhibit isolated point singularities in the interior of $Ω$ or corner and edge singularities at the boundary $\partial Ω$. The exponential expression rate bounds proved here imply uniform exponential expressivity by ReLU NNs of solution families for several elliptic boundary and eigenvalue problems with analytic data. The exponential approximation rates are shown to hold in space dimension $d = 2$ on Lipschitz polygons with straight sides, and in space dimension $d=3$ on Fichera-type polyhedral domains with plane faces. The constructive proofs indicate in particular that NN depth and size increase poly-logarithmically with respect to the target NN approximation accuracy $\varepsilon>0$ in $H^1(Ω)$. The results cover in particular solution sets of linear, second order elliptic PDEs with analytic data and certain nonlinear elliptic eigenvalue problems with analytic nonlinearities and singular, weighted analytic potentials as arise in electron structure models. In the latter case, the functions correspond to electron densities that exhibit isolated point singularities at the positions of the nuclei. Our findings provide in particular mathematical foundation of recently reported, successful uses of deep neural networks in variational electron structure algorithms.

Motivation & Objective

• To establish provably exponential approximation rates for ReLU neural networks in Sobolev spaces for functions with point and edge singularities.
• To provide a mathematical foundation for the success of deep neural networks in approximating solutions to elliptic PDEs with analytic data and singularities.
• To analyze the dependence of network depth and size on approximation accuracy for singular functions in $ H^1(\Omega) $.
• To extend exponential expressivity results to weighted analytic function classes on polygonal and polyhedral domains with corner and edge singularities.
• To demonstrate that solution families of linear and nonlinear elliptic PDEs with analytic data can be uniformly and exponentially approximated by ReLU networks.

Proposed method

• Constructs a ReLU neural network realization that approximates univariate piecewise polynomial functions with singularities using depth- and width-efficient architectures.
• Employs tensor product $ hp $-finite element spaces on geometric meshes to construct high-order approximations of weighted analytic functions in $ Q = (0,1)^d $, $ d=2,3 $.
• Uses a constructive proof strategy to embed local approximations into global ReLU networks via concatenation, parallelization, and emulation of identity and multiplication operations.
• Applies weighted Sobolev spaces of Kondrat’ev type to model analytic regularity with corner and edge singularities in polytopal domains.
• Establishes exponential convergence by bounding the $ H^1 $-norm error of the network approximation in terms of the number of parameters $ M $, showing decay $ \mathcal{O}(\exp(-bM^{1/(2d+1)})) $.
• Applies the general approximation result to specific PDEs, including nonlinear eigenvalue problems with singular potentials and second-order elliptic problems with analytic data.

Experimental results

Research questions

• RQ1 Can ReLU neural networks achieve exponential approximation rates for functions with isolated point and edge singularities in 2D and 3D domains?
• RQ2 What is the dependence of network depth and size on the approximation accuracy $ \varepsilon $ for singular functions in $ H^1(\Omega) $?
• RQ3 Can deep ReLU networks uniformly and exponentially approximate solution families of elliptic PDEs with analytic data and singularities?
• RQ4 How do the approximation rates of ReLU networks compare to classical finite element methods for singular solutions?
• RQ5 What is the theoretical basis for the observed success of deep learning in variational electron structure calculations with singular electron densities?

Key findings

• The paper proves exponential approximation rates of the form $ \|u - u_N\|_{H^1(\Omega)} \leq C\exp(-bM^{1/(2d+1)}) $ for $ d=2,3 $, where $ M $ is the number of parameters in the ReLU network.
• The approximation error decays exponentially with respect to the number of network parameters, with the exponent $ 1/(2d+1) $, indicating poly-logarithmic growth in depth and size with respect to accuracy.
• The results apply to weighted analytic function classes on Lipschitz polygons (2D) and Fichera-type polyhedral domains (3D), which include solutions to elliptic PDEs with corner and edge singularities.
• The method constructs ReLU networks that achieve exponential convergence by leveraging tensor product $ hp $-finite element approximations and neural network operations such as concatenation and multiplication emulation.
• The framework supports approximation of solutions to nonlinear eigenvalue problems with analytic nonlinearities and singular, weighted analytic potentials, such as electron densities with point singularities at nuclei.
• The theoretical results provide a mathematical justification for the empirical success of deep neural networks in variational electron structure algorithms and PDE solvers with singular solutions.