[Paper Review] An overview on deep learning-based approximation methods for partial differential equations

Christian Beck, Martin Hutzenthaler+2 more|arXiv (Cornell University)|Dec 22, 2020

Model Reduction and Neural Networks31 citations

TL;DR

The paper surveys how deep learning approaches approximate high-dimensional PDEs, detailing linear and nonlinear methods (like deep Galerkin and deep splitting), theoretical insights, simulations, and available code.

ABSTRACT

It is one of the most challenging problems in applied mathematics to approximatively solve high-dimensional partial differential equations (PDEs). Recently, several deep learning-based approximation algorithms for attacking this problem have been proposed and tested numerically on a number of examples of high-dimensional PDEs. This has given rise to a lively field of research in which deep learning-based methods and related Monte Carlo methods are applied to the approximation of high-dimensional PDEs. In this article we offer an introduction to this field of research by revisiting selected mathematical results related to deep learning approximation methods for PDEs and reviewing the main ideas of their proofs. We also provide a short overview of the recent literature in this area of research.

Motivation & Objective

• Introduce the challenge of solving high-dimensional PDEs and motivate learning-based approaches.
• Review and distill key deep learning-based methods for linear PDEs, especially linear Kolmogorov (heat) equations.
• Present and outline nonlinear PDE techniques with two main methods: the deep Galerkin method and the deep splitting method.
• Provide theoretical context and discuss partial results on overcoming the curse of dimensionality in PDE approximation.
• Offer guidance on simulations and provide source code to illustrate practical implementations.

Proposed method

• Formulate linear PDEs as infinite-dimensional stochastic optimization problems via Feynman–Kac representations.
• Describe deep neural networks and their realizations as parameterized function approximators for solving PDEs.
• Outline a concrete deep learning-based scheme for linear PDEs by minimizing an expected squared loss over stochastic samples.
• Detail two nonlinear PDE approaches: the deep Galerkin method and the deep splitting method, with corresponding loss functionals.
• Provide simple PyTorch implementations to illustrate the methods and discuss extensions to more general Kolmogorov PDEs.

Experimental results

Research questions

• RQ1 How can linear and nonlinear PDEs be recast as stochastic optimization problems suitable for neural network approximation?
• RQ2 Can deep neural networks approximate PDE solutions efficiently in high dimensions, potentially overcoming the curse of dimensionality?
• RQ3 What are the theoretical guarantees and limitations of deep learning-based PDE approximation methods?
• RQ4 How do the deep Galerkin and deep splitting methods compare conceptually and practically for semilinear/nonlinear PDEs?
• RQ5 What are practical implementations and simulations that demonstrate these methods on high-dimensional problems?

Key findings

• Deep learning-based approximations can reformulate PDEs into stochastic optimization problems whose solutions correspond to PDE solutions.
• DNNs can serve as flexible function classes to approximate PDE solutions over high-dimensional domains.
• The deep Galerkin method provides a framework to solve semilinear PDEs by minimizing a loss that encodes the PDE and terminal condition, linking optimization to PDE satisfaction.
• The deep splitting method and other approaches extend neural PDE solvers to nonlinear settings, providing practical algorithms and accompanying theory.
• Simulations and code illustrate feasibility and performance on high-dimensional problems, while acknowledging that full theoretical justification for overcoming the curse of dimensionality remains partial.
• The article includes explicit PyTorch examples to aid replication of the methods.