[Paper Review] Deep neural network approximation for high-dimensional parabolic Hamilton-Jacobi-Bellman equations

Philipp Grohs, Lukas Herrmann|arXiv (Cornell University)|Mar 9, 2021

Model Reduction and Neural Networks4 citations

TL;DR

This paper demonstrates that deep neural networks (DNNs) can approximate solutions to high-dimensional parabolic Hamilton-Jacobi-Bellman (HJB) equations without suffering the curse of dimensionality, under conditions where the dynamics are affine in controls and the cost is quadratic in controls with bounded control values. The key result is a constructive proof showing that ReLU and ReCU DNNs can achieve this approximation with size growing polynomially in the inverse accuracy and dimension, enabling efficient numerical solution of high-dimensional stochastic optimal control problems.

ABSTRACT

The approximation of solutions to second order Hamilton--Jacobi--Bellman (HJB) equations by deep neural networks is investigated. It is shown that for HJB equations that arise in the context of the optimal control of certain Markov processes the solution can be approximated by deep neural networks without incurring the curse of dimension. The dynamics is assumed to depend affinely on the controls and the cost depends quadratically on the controls. The admissible controls take values in a bounded set.

Motivation & Objective

• To address the curse of dimensionality in solving high-dimensional parabolic HJB equations arising in stochastic optimal control.
• To establish that deep neural networks can approximate the value function and its gradient for such equations without exponential cost in dimension.
• To provide a constructive proof of DNN approximation with polynomial dependence on dimension and accuracy, under specific structural assumptions on the control problem.
• To extend existing DNN approximation theory to non-linear PDEs with gradient-dependent non-linearities, which are central to HJB equations.
• To validate the feasibility of using DNNs for practical high-dimensional optimal control by proving approximation error bounds independent of dimension.

Proposed method

• The authors use a combination of probabilistic representation of the HJB solution via the Feynman-Kac formula and stochastic control theory to express the value function as an expectation over controlled diffusion processes.
• They apply a localization technique using a smooth cutoff function χ to restrict the solution to local balls, enabling the use of local regularity estimates.
• The proof relies on Sobolev and Hölder regularity theory for parabolic PDEs, showing that the localized solution and its derivatives are Hölder continuous in space-time.
• A mollification procedure is applied to the right-hand side of the PDE to construct a sequence of smooth approximations, ensuring convergence in Hölder norms.
• The convergence of the mollified solutions in $C^{1,2}$-Hölder spaces is established via regularity estimates, proving that the original solution inherits $C^{1,2}$-regularity at every point.
• The final construction of the DNN approximation leverages the fact that ReLU and ReCU networks can represent Hölder continuous functions with size growing polynomially in the inverse accuracy and dimension.

Experimental results

Research questions

• RQ1 Can deep neural networks approximate solutions to high-dimensional parabolic HJB equations without incurring the curse of dimensionality?
• RQ2 Does the structure of affine dynamics and quadratic cost in the control variable allow for dimension-independent approximation rates using DNNs?
• RQ3 Can the non-linearity in the HJB equation, which depends on the gradient of the value function, be effectively captured by ReLU and ReCU deep neural networks?
• RQ4 Is it possible to construct DNNs whose size grows polynomially in the inverse accuracy and dimension, rather than exponentially, for such HJB problems?
• RQ5 Can the regularity of the value function and its gradient be leveraged to build a constructive DNN approximation scheme with provable error bounds?

Key findings

• The value function of the HJB equation is shown to be $C^{1,2}$-regular in space-time under the given assumptions, enabling the use of smooth approximation techniques.
• The solution to the HJB equation can be approximated by deep neural networks with ReLU and ReCU activation functions, achieving approximation error $\varepsilon$ with network size growing polynomially in $\varepsilon^{-1}$ and dimension $d$, avoiding the exponential cost of classical methods.
• The proof establishes that the approximation is independent of the spatial dimension $d$ in terms of exponential scaling, thus overcoming the curse of dimensionality.
• The construction relies on a sequence of mollified PDEs with smooth data, whose solutions converge in $C^{1,2}$-Hölder norms, ensuring the existence of a smooth limit.
• The final DNN approximation is built by composing and adding ReLU and ReCU networks, which can represent the required Hölder continuous functions with controlled size.
• The result confirms that deep learning methods are theoretically viable for high-dimensional stochastic optimal control problems under affine and quadratic structural assumptions.