[Paper Review] On the approximation of the solution of partial differential equations by artificial neural networks trained by a multilevel Levenberg-Marquardt method

Henri Calandra, Serge Gratton+2 more|arXiv (Cornell University)|Apr 9, 2019

Model Reduction and Neural Networks49 references4 citations

TL;DR

This paper proposes a multilevel Levenberg-Marquardt method to train feedforward neural networks for approximating solutions of partial differential equations (PDEs). By formulating the problem as a least squares optimization using the PDE residual as the loss function, and introducing a heuristic algebraic multigrid-inspired hierarchy for transfer operators, the method achieves faster convergence and reduced computational cost—up to a 2x reduction in floating-point operations—without sacrificing solution accuracy compared to standard one-level training.

ABSTRACT

This paper is concerned with the approximation of the solution of partial differential equations by means of artificial neural networks. Here a feedforward neural network is used to approximate the solution of the partial differential equation. The learning problem is formulated as a least squares problem, choosing the residual of the partial differential equation as a loss function, whereas a multilevel Levenberg-Marquardt method is employed as a training method. This setting allows us to get further insight into the potential of multilevel methods. Indeed, when the least squares problem arises from the training of artificial neural networks, the variables subject to optimization are not related by any geometrical constraints and the standard interpolation and restriction operators cannot be employed any longer. A heuristic, inspired by algebraic multigrid methods, is then proposed to construct the multilevel transfer operators. Numerical experiments show encouraging results related to the efficiency of the new multilevel optimization method for the training of artificial neural networks, compared to the standard corresponding one-level procedure.

Motivation & Objective

• To address the challenge of training artificial neural networks (ANNs) for high-dimensional or highly nonlinear PDEs, where standard optimization methods suffer from slow convergence and high computational cost.
• To extend multigrid principles—typically used for solving PDEs—to the training of ANNs by constructing a hierarchy of optimization problems.
• To develop a heuristic method for defining multilevel transfer operators in the absence of geometric structure in ANN weights, inspired by algebraic multigrid (AMG) techniques.
• To evaluate whether multilevel optimization can outperform standard one-level Levenberg-Marquardt training in terms of convergence speed and computational efficiency for PDE solution approximation.
• To explore the feasibility of applying multilevel optimization to non-geometric, large-scale least squares problems arising in ANN training.

Proposed method

• A feedforward neural network is used to approximate the solution of a PDE, with the residual of the PDE serving as the loss function in a nonlinear least squares problem.
• The training problem is solved using a multilevel Levenberg-Marquardt (MLM) method, which leverages a hierarchy of coarser optimization problems to accelerate convergence.
• A heuristic transfer operator construction is proposed, inspired by algebraic multigrid (AMG), to map between fine and coarse network weight spaces despite the absence of geometric constraints.
• The method is applied to both 1D and 2D PDEs, including Helmholtz and nonlinear elliptic equations, with varying network widths and PDE parameters.
• The performance of the MLM method is compared against the standard one-level Levenberg-Marquardt (LM) method in terms of iterations, RMSE, and computational cost (floating-point operations).
• The approach is generalizable to other least squares problems where the solution is represented by a neural network, including ODEs and inverse problems.

Experimental results

Research questions

• RQ1 Can multilevel optimization techniques, traditionally used for solving PDEs, be effectively adapted to train artificial neural networks for PDE solution approximation?
• RQ2 How can effective multilevel transfer operators be constructed when the optimization variables (network weights) lack geometric structure?
• RQ3 Does the proposed multilevel Levenberg-Marquardt method converge faster and with lower computational cost than the standard one-level method for PDE-related training problems?
• RQ4 To what extent does the multilevel approach preserve solution accuracy compared to one-level training, especially for complex or high-dimensional PDEs?
• RQ5 Can the method be extended to deeper or multilayer neural networks, and what challenges arise from increased nonlinearity?

Key findings

• The multilevel Levenberg-Marquardt (MLM) method required fewer iterations than the one-level method for most test cases, particularly for problems with higher nonlinearity (e.g., ν=20).
• Despite performing more iterations in some cases, the MLM method reduced the total number of floating-point operations by a factor of up to 5.3 for 1D problems and 1.8 for 2D problems, with an average gain of approximately 2x.
• The root-mean-square error (RMSE) of the solution remained comparable between MLM and one-level training, with values of 10⁻⁵ and 10⁻³ for ν=20 and ν=1 respectively, indicating no loss in accuracy.
• For the 2D Helmholtz equation with Dirichlet conditions, the method achieved a reasonable approximation in fewer than 200 iterations, even when further iterations failed to improve accuracy.
• The method demonstrated robustness across different PDE types and network sizes, with performance gains increasing for more complex problems requiring higher network width (r=2⁹).
• The heuristic transfer operator construction based on algebraic multigrid principles proved effective in creating a functional multilevel hierarchy, even without geometric constraints on the weights.