[Paper Review] Solving and Learning Nonlinear PDEs with Gaussian Processes

Yifan Chen, Bamdad Hosseini+2 more|arXiv (Cornell University)|Mar 24, 2021

Gaussian Processes and Bayesian Inference82 references21 citations

TL;DR

This paper introduces a novel Gaussian process framework for solving nonlinear PDEs and inverse problems by formulating the solution as a maximum a posteriori (MAP) estimator under a GP prior, constrained via collocation. It achieves provable convergence and efficient computation via a finite-dimensional reduction using derivative variables, enabling robust, meshless solution of diverse PDEs with guaranteed error bounds and low iteration counts (2–10).

ABSTRACT

We introduce a simple, rigorous, and unified framework for solving nonlinear partial differential equations (PDEs), and for solving inverse problems (IPs) involving the identification of parameters in PDEs, using the framework of Gaussian processes. The proposed approach: (1) provides a natural generalization of collocation kernel methods to nonlinear PDEs and IPs; (2) has guaranteed convergence for a very general class of PDEs, and comes equipped with a path to compute error bounds for specific PDE approximations; (3) inherits the state-of-the-art computational complexity of linear solvers for dense kernel matrices. The main idea of our method is to approximate the solution of a given PDE as the maximum a posteriori (MAP) estimator of a Gaussian process conditioned on solving the PDE at a finite number of collocation points. Although this optimization problem is infinite-dimensional, it can be reduced to a finite-dimensional one by introducing additional variables corresponding to the values of the derivatives of the solution at collocation points; this generalizes the representer theorem arising in Gaussian process regression. The reduced optimization problem has the form of a quadratic objective function subject to nonlinear constraints; it is solved with a variant of the Gauss--Newton method. The resulting algorithm (a) can be interpreted as solving successive linearizations of the nonlinear PDE, and (b) in practice is found to converge in a small number of iterations (2 to 10), for a wide range of PDEs. Most traditional approaches to IPs interleave parameter updates with numerical solution of the PDE; our algorithm solves for both parameter and PDE solution simultaneously. Experiments on nonlinear elliptic PDEs, Burgers' equation, a regularized Eikonal equation, and an IP for permeability identification in Darcy flow illustrate the efficacy and scope of our framework.

Motivation & Objective

• To develop a unified, theoretically grounded framework for solving nonlinear PDEs and inverse problems using Gaussian processes.
• To generalize kernel collocation methods to nonlinear PDEs while preserving convergence guarantees and computational efficiency.
• To enable simultaneous solution of PDEs and identification of unknown parameters in inverse problems within a single optimization framework.
• To provide a path to compute rigorous error bounds for PDE approximations using the proposed GP-based approach.
• To leverage the computational efficiency of dense kernel matrix solvers while maintaining state-of-the-art complexity for nonlinear problems.

Proposed method

• Formulate the solution of a nonlinear PDE as the MAP estimator of a Gaussian process conditioned on satisfying the PDE at a finite set of collocation points.
• Reduce the infinite-dimensional optimization problem to a finite-dimensional one by introducing variables for solution and derivative values at collocation points, generalizing the representer theorem.
• Construct a quadratic objective function with nonlinear constraints derived from the PDE and boundary conditions, solved via a Gauss–Newton variant.
• Interpret the algorithm as solving successive linearizations of the nonlinear PDE, ensuring fast convergence in 2–10 iterations across diverse problems.
• Use the resulting GP approximation to compute error bounds via the dual space and embedding inequalities, leveraging the regularity of the solution and fill distance of collocation points.
• Enable kernel learning (e.g., via MLE, MAP, or cross-validation) to adapt the kernel to the solution space, improving accuracy for irregular or nonlinear PDEs.

Experimental results

Research questions

• RQ1 Can a Gaussian process framework be rigorously extended to solve nonlinear PDEs with provable convergence and error control?
• RQ2 How can the solution of a nonlinear PDE and the identification of unknown parameters in inverse problems be simultaneously optimized within a probabilistic framework?
• RQ3 What is the computational complexity of solving nonlinear PDEs using GP-based collocation, and can it match that of linear solvers for dense kernel matrices?
• RQ4 How do kernel choice and regularity of the solution affect the accuracy and convergence of the GP-based PDE solver?
• RQ5 Can the framework be generalized to non-Gaussian priors while preserving computational efficiency and theoretical guarantees?

Key findings

• The proposed method achieves convergence for a very general class of nonlinear PDEs, with convergence rates bounded by the fill distance of collocation points and the solution's Sobolev regularity.
• The algorithm converges in only 2 to 10 iterations across a range of PDEs, including nonlinear elliptic equations, Burgers’ equation, and a regularized Eikonal equation.
• Error bounds are derived via the dual space and embedding inequalities, showing that the approximation error decays as $ O(h^s) $, where $ h $ is the fill distance and $ s $ relates to the solution's Sobolev norm.
• The method inherits the computational complexity of state-of-the-art solvers for dense kernel matrices, enabling efficient solution of large-scale problems.
• Kernel learning via MLE, MAP, or cross-validation is feasible and improves accuracy, especially for irregular solutions or nonlinear PDEs.
• The framework naturally extends to non-Gaussian priors through nonlinear transformations of Gaussian processes, enabling new approaches to Bayesian inverse problems with complex priors.