[Paper Review] A Local Deep Learning Method for Solving High Order Partial Differential Equations

Quanhui Zhu, Yang Jiang|arXiv (Cornell University)|Mar 16, 2021

Model Reduction and Neural Networks7 citations

TL;DR

This paper proposes a local deep learning method that reformulates high-order PDEs into systems of first-order equations using intermediate variables, enabling stable and efficient training via multi-output neural networks without computing high-order derivatives. The method achieves superior convergence and robustness in high-dimensional problems compared to standard deep Galerkin methods.

ABSTRACT

At present, deep learning based methods are being employed to resolve the computational challenges of high-dimensional partial differential equations (PDEs). But the computation of the high order derivatives of neural networks is costly, and high order derivatives lack robustness for training purposes. We propose a novel approach to solving PDEs with high order derivatives by simultaneously approximating the function value and derivatives. We introduce intermediate variables to rewrite the PDEs into a system of low order differential equations as what is done in the local discontinuous Galerkin method. The intermediate variables and the solutions to the PDEs are simultaneously approximated by a multi-output deep neural network. By taking the residual of the system as a loss function, we can optimize the network parameters to approximate the solution. The whole process relies on low order derivatives. Numerous numerical examples are carried out to demonstrate that our local deep learning is efficient, robust, flexible, and is particularly well-suited for high-dimensional PDEs with high order derivatives.

Motivation & Objective

• To address the computational infeasibility and instability of computing high-order derivatives in deep learning for PDEs.
• To overcome the parameter scaling problem and training instability in high-order PDEs, especially for phase field equations like Cahn-Hilliard.
• To develop a robust, scalable deep learning framework for high-dimensional PDEs with high-order derivatives.
• To improve training efficiency and convergence by reducing derivative order through variable substitution.

Proposed method

• Rewrite the original high-order PDE into a system of first-order PDEs by introducing intermediate variables for each derivative order.
• Use a multi-output deep neural network to simultaneously approximate the solution and all intermediate variables.
• Formulate the loss function as the L2 residual of the entire system, avoiding direct computation of high-order derivatives.
• Train the network using standard optimization with automatic differentiation on only first-order derivatives.
• Leverage the structure of the local discontinuous Galerkin (LDG) method to guide variable introduction and system formulation.
• Apply the method to high-dimensional PDEs, including heat equations and phase field models, with adaptive learning rates and network width adjustments.

Experimental results

Research questions

• RQ1 Can high-order PDEs be solved efficiently using deep learning without computing high-order derivatives?
• RQ2 Does introducing intermediate variables improve training stability and convergence in high-dimensional PDEs?
• RQ3 How does the proposed method compare to standard deep Galerkin methods in terms of accuracy and computational cost?
• RQ4 Can the method handle challenging problems like the Cahn-Hilliard equation with small parameter ε?
• RQ5 Does the reduction of derivative order mitigate gradient vanishing or exploding issues in deep learning for PDEs?

Key findings

• The local deep Galerkin method (LDGM) achieves stable and fast convergence in solving a 5D heat equation, while the standard deep Galerkin method (DGM) fails to converge after 50,000 iterations.
• The LDGM shows significantly faster error decay in training, with relative L2 error dropping rapidly even with limited network capacity.
• For the 5D heat equation, the LDGM reduces the relative L2 error by over 90% compared to DGM at 50,000 iterations, with a 50% reduction in computation time.
• The method successfully solves the Cahn-Hilliard equation, which previous deep learning methods failed to converge on due to high-order derivatives and small ε.
• The use of intermediate variables eliminates the need for high-order automatic differentiation, reducing memory and computation cost by up to 80% in tested configurations.
• Increasing network depth, width, and sampling nodes further improves accuracy, indicating scalability and adaptability to complex domains.