[Paper Review] Adaptive-Growth Randomized Neural Networks for PDEs: Algorithms and Numerical Analysis

Haoning Dang, Fei Wang+1 more|arXiv (Cornell University)|Aug 30, 2024

Neural Networks and Applications4 citations

TL;DR

This paper proposes Adaptive-Growing Randomized Neural Networks (AG-RNN) to solve PDEs with sharp or discontinuous solutions by dynamically expanding network width and depth using frequency-based initialization, residual-driven growth, and domain splitting. The method achieves high accuracy with minimal training by constructively adding neurons and layers without retraining, outperforming traditional methods in challenging PDEs with discontinuities or high-frequency components.

ABSTRACT

Randomized neural network (RaNN) methods have been proposed for solving various partial differential equations (PDEs), demonstrating high accuracy and efficiency. However, initializing the fixed parameters remains challenging. Additionally, RaNNs often struggle to approximate PDE solutions with sharp gradients or discontinuities when using smooth activations and shallow architectures. In this paper, we propose an Adaptive-Growth Randomized Neural Network (AG-RaNN) to address these challenges. We first design a frequency-based initialization for a shallow RaNN. Using the residual as an error indicator, we then adaptively grow the network in width (neuron growth) and depth (layer growth) to improve the accuracy of the numerical solution. The weights and biases of new neurons are constructed rather than trained, which enhances the approximation power without additional nonlinear optimization. To handle discontinuities, we further introduce a domain splitting strategy. We also establish a unified error analysis covering approximation, statistical, and optimization errors. Extensive numerical experiments demonstrate the efficiency and accuracy of AG-RaNN.

Motivation & Objective

• Address the challenge of poor initialization and limited approximation capacity in randomized neural networks (RNNs) for solving PDEs.
• Overcome the difficulty of solving PDEs with sharp gradients or discontinuous solutions using standard RNNs.
• Develop a mesh-free, adaptive framework that improves solution accuracy by dynamically growing network width and depth without retraining.
• Introduce domain splitting to handle discontinuous solutions more effectively than single-network approaches.
• Achieve high accuracy and efficiency in solving complex PDEs, including those with high-frequency oscillations or shock waves.

Proposed method

• Use frequency information from the Fourier transform of the PDE’s source term to guide the initialization of network weights and biases, ensuring the initial network captures dominant solution frequencies.
• Apply a residual-based error indicator to detect regions of high error and guide neuron growth in existing hidden layers.
• Constructively add new neurons with adaptive weights and biases using a frequency-aware strategy, avoiding backpropagation and enabling immediate improvement in approximation.
• Implement layer growth by adding new hidden layers with optimized parameters, enhancing the network’s ability to model complex, non-smooth solution patterns.
• Apply domain splitting by dividing the computational domain into subdomains, each solved by a separate AG-RNN, to handle discontinuities more effectively.
• Use an $L^1$-norm stopping criterion to control growth, halting expansion when solution changes fall below a threshold ($varepsilon_s = 10^{-4}$).

Experimental results

Research questions

• RQ1 Can frequency-based parameter initialization improve the initial approximation quality of randomized neural networks for PDEs?
• RQ2 Can adaptive neuron growth without retraining significantly enhance solution accuracy in PDEs with complex or oscillatory solutions?
• RQ3 Does layer growth improve the network’s ability to capture multi-scale or non-smooth features in PDE solutions?
• RQ4 Can domain splitting effectively handle PDEs with discontinuous or shock-like solutions where standard RNNs fail?
• RQ5 How does the AG-RNN framework compare in accuracy and efficiency to traditional methods like finite element or ELM for challenging PDEs?

Key findings

• The frequency-based initialization strategy significantly improves the initial approximation quality, reducing the initial relative $L^2$ error to 1.8459e-01 in Example 6.
• Neuron growth with $lambda=100$ and $r_{\text{max}}=100$ reduced the relative $L^2$ error from 1.8459e-01 to 1.2029e-01 after three growth stages, demonstrating progressive accuracy improvement.
• Layer growth added a new hidden layer with $m_2=600$ neurons, reducing the relative $L^2$ error to 1.1055e-01, though effectiveness was limited by the continuity of activation functions in discontinuous regions.
• Domain splitting with $R=2$ subdomains achieved the lowest relative $L^2$ error of 3.4999e-02, outperforming both neuron and layer growth strategies in the shock wave case.
• The AG-RNN framework achieved superior accuracy compared to standard RNNs and finite element methods, particularly in problems with discontinuities or high-frequency components.
• The $L^1$-norm stopping criterion effectively controlled growth, preventing overfitting and ensuring convergence when solution changes fell below $10^{-4}$.