[Paper Review] Quasi-Monte Carlo sampling for machine-learning partial differential equations

Jingrun Chen, Rui Du+2 more|arXiv (Cornell University)|Nov 5, 2019

Mathematical Approximation and Integration21 references4 citations

TL;DR

This paper proposes replacing Monte Carlo (MC) sampling with quasi-Monte Carlo (QMC) sampling in deep learning-based solvers for high-dimensional partial differential equations (PDEs), specifically within the deep Ritz method framework. By using low-discrepancy sequences instead of random samples, QMC reduces training data requirements by over two orders of magnitude for the same accuracy, while theoretical analysis confirms convergence with a rate proportional to QMC integration error.

ABSTRACT

Solving partial differential equations in high dimensions by deep neural network has brought significant attentions in recent years. In many scenarios, the loss function is defined as an integral over a high-dimensional domain. Monte-Carlo method, together with the deep neural network, is used to overcome the curse of dimensionality, while classical methods fail. Often, a deep neural network outperforms classical numerical methods in terms of both accuracy and efficiency. In this paper, we propose to use quasi-Monte Carlo sampling, instead of Monte-Carlo method to approximate the loss function. To demonstrate the idea, we conduct numerical experiments in the framework of deep Ritz method proposed by Weinan E and Bing Yu. For the same accuracy requirement, it is observed that quasi-Monte Carlo sampling reduces the size of training data set by more than two orders of magnitude compared to that of MC method. Under some assumptions, we prove that quasi-Monte Carlo sampling together with the deep neural network generates a convergent series with rate proportional to the approximation accuracy of quasi-Monte Carlo method for numerical integration. Numerically the fitted convergence rate is a bit smaller, but the proposed approach always outperforms Monte Carlo method. It is worth mentioning that the convergence analysis is generic whenever a loss function is approximated by the quasi-Monte Carlo method, although observations here are based on deep Ritz method.

Motivation & Objective

• Address the curse of dimensionality in solving high-dimensional PDEs using deep neural networks.
• Overcome the inefficiency of Monte Carlo sampling in approximating high-dimensional integrals within machine learning PDE solvers.
• Demonstrate that quasi-Monte Carlo (QMC) sampling can significantly reduce training data size while maintaining or improving accuracy.
• Provide a theoretical convergence analysis for QMC-based deep learning solvers under strong convexity assumptions.
• Establish a general framework applicable beyond the deep Ritz method, including to other methods like deep Galerkin.

Proposed method

• Replace standard Monte Carlo sampling with quasi-Monte Carlo (QMC) sampling to approximate the loss function integral in high-dimensional domains.
• Use low-discrepancy sequences (e.g., Sobol’ sequences) instead of i.i.d. random samples to reduce integration error.
• Integrate QMC sampling into the deep Ritz method, where the loss function is a variational form of the PDE.
• Apply stochastic gradient descent (SGD) with a fixed stepsize to optimize the deep neural network parameters.
• Analyze convergence using a recursive inequality involving the expected error and the sampling error term $ r(N) $, which quantifies QMC integration error.
• Derive a theoretical bound showing that the expected error decays linearly with iteration count and is proportional to the QMC error rate $ r(N) $.

Experimental results

Research questions

• RQ1 Can quasi-Monte Carlo sampling reduce the training data size required to achieve a given accuracy in deep learning-based PDE solvers?
• RQ2 How does the convergence rate of a deep neural network solver based on QMC compare to that of Monte Carlo under the same conditions?
• RQ3 What is the theoretical convergence behavior of a deep learning PDE solver when using QMC sampling for loss function approximation?
• RQ4 Does the performance advantage of QMC over MC persist when the loss function is non-convex, despite the theoretical assumptions relying on strong convexity?
• RQ5 To what extent is the proposed QMC-based method generalizable to other machine learning PDE solvers beyond the deep Ritz method?

Key findings

• For the same accuracy requirement, quasi-Monte Carlo sampling reduces the required training data size by more than two orders of magnitude compared to Monte Carlo sampling.
• Numerical experiments show that the convergence rate of the QMC-based method is slightly lower than the theoretical bound but consistently outperforms Monte Carlo in both accuracy and efficiency.
• Theoretical analysis proves that the expected error of the deep neural network solution converges linearly with iteration count and is proportional to the QMC integration error rate $ r(N) $, under strong convexity assumptions.
• When the loss function is strongly convex, the expected error converges to $ \frac{\alpha L C_V}{2c\mu} r(N) $, which tends to zero as $ N \to \infty $, indicating convergence.
• Even under non-convex conditions, QMC sampling still outperforms MC in practice, suggesting robustness beyond the theoretical assumptions.
• The proposed method is general and applicable to other machine learning PDE solvers, such as the deep Galerkin method, wherever high-dimensional integrals are approximated via sampling.