[Paper Review] Extreme learning machine collocation for the numerical solution of elliptic PDEs with sharp gradients

Francesco Calabrò, Gianluca Fabiani+1 more|arXiv (Cornell University)|Dec 10, 2020

Model Reduction and Neural Networks61 references79 citations

TL;DR

This paper proposes an extreme learning machine (ELM) collocation method for solving 1D elliptic PDEs with sharp gradients, such as boundary layers, by fixing internal weights and biases of a single-hidden-layer feedforward network with sigmoidal activation. The method avoids network training by solving a linear least-squares problem, achieving high accuracy and computational efficiency, outperforming finite differences in both error and cost for problems with steep gradients.

ABSTRACT

We introduce a new numerical method based on machine learning to approximate the solution of elliptic partial differential equations with collocation using a set of sigmoidal functions. We show that a feedforward neural network with a single hidden layer with sigmoidal functions and fixed, random, internal weights and biases can be used to compute accurately a collocation solution. The choice to fix internal weights and bias leads to the so-called Extreme Learning Machine network. We discuss how to determine the range for both internal weights and biases in order to obtain a good underlining approximating space, and we explore the required number of collocation points. We demonstrate the efficiency of the proposed method with several one-dimensional diffusion-advection-reaction problems that exhibit steep behaviors, such as boundary layers. The boundary conditions are imposed directly as collocation equations. We point out that there is no need of training the network, as the proposed numerical approach results to a linear problem that can be easily solved using least-squares. Numerical results show that the proposed method achieves a good accuracy. Finally, we compare the proposed method with finite differences and point out the significant improvements in terms of computational cost, thus avoiding the time-consuming training phase.

Motivation & Objective

• Address the challenge of numerically solving elliptic PDEs with sharp gradients, such as boundary layers, where classical methods like finite differences suffer from instability and oscillations.
• Explore the applicability of Extreme Learning Machines (ELM) for solving differential equations, particularly in cases where traditional ELM approaches have failed due to poor approximation of steep gradients.
• Develop a collocation-based ELM framework that fixes internal weights and biases a priori, eliminating the need for network training while maintaining high accuracy.
• Demonstrate the method's robustness and efficiency on benchmark problems with known analytical solutions, including those with comb-like profiles and high Péclet numbers.
• Compare the proposed ELM collocation method with high-order finite difference methods to quantify improvements in accuracy and computational cost.

Proposed method

• Use a single-hidden-layer feedforward neural network with sigmoidal activation functions σ(x) = 1 / (1 + exp(−x)) and fixed, randomly chosen internal weights αi and biases βi.
• Construct the solution as a linear combination of sigmoidal functions: G(x) = Σ wi σ(αi x + βi), where only the output weights wi are learned via collocation.
• Impose boundary conditions directly as collocation equations at boundary points, ensuring exact satisfaction of Dirichlet conditions.
• Formulate the numerical solution as a linear system of equations by enforcing the PDE at M collocation points, resulting in an underdetermined system solved via least-squares.
• Determine the range of internal weights and biases a priori to ensure sufficient approximation power and coverage of the solution's sharp gradient regions.
• Use an underdetermined collocation scheme (M < n) to avoid ill-conditioning and improve numerical stability, especially for large n.

Experimental results

Research questions

• RQ1 Can ELM networks with fixed internal weights and biases approximate solutions of elliptic PDEs with sharp gradients more effectively than traditional methods?
• RQ2 How does the choice of internal weight and bias ranges affect the approximation quality of the ELM-based collocation method?
• RQ3 What is the optimal number of collocation points M relative to the number of neurons n for achieving high accuracy without ill-conditioning?
• RQ4 How does the ELM collocation method compare in accuracy and computational cost to high-order finite difference methods for problems with boundary layers?
• RQ5 Can the ELM collocation method capture highly oscillatory or comb-like solutions with minimal error using a fixed, non-trainable network architecture?

Key findings

• The proposed ELM collocation method achieves high accuracy in solving 1D elliptic PDEs with boundary layers, with L2 errors below 10−10 for problems with Péclet numbers up to 500.
• For the comb-like solution with 10 zeros and 5 oscillations (ε = 1/(10π)), the method achieved L∞ errors below 10−6 with n = 1280 neurons and M = n/2 collocation points.
• The method outperforms high-order finite difference methods (FDM) in both error and computational cost, especially for high Péclet numbers and problems with steep gradients.
• The ELM solution is C∞ smooth, in contrast to FDM solutions, which are typically piecewise polynomial and less smooth.
• The method remains stable and accurate even when the number of collocation points M is reduced to M = n/2, demonstrating robustness to underdetermination.
• The absence of network training significantly reduces computational cost compared to standard ANN-based solvers, making it highly efficient for problems requiring repeated or fast solutions.