[Paper Review] Meshless discretization of LQ-type stochastic control problems

Ralf Banisch, Carsten Hartmann|arXiv (Cornell University)|Sep 28, 2013

Markov Chains and Monte Carlo Methods39 references3 citations

TL;DR

This paper introduces a meshless Galerkin discretization scheme for linear-quadratic (LQ)-type stochastic control problems on indefinite time horizons, leveraging a logarithmic transformation to convert the problem into a linear boundary value problem. The method achieves strong L² error bounds and is dual to a Markov decision problem, with applications to high-dimensional systems like molecular dynamics using committor function bases.

ABSTRACT

We propose a novel Galerkin discretization scheme for stochastic optimal control problems on an indefinite time horizon. The control problems are linear-quadratic in the controls, but possibly nonlinear in the state variables, and the discretization is based on the fact that problems of this kind can be transformed into linear boundary value problems by a logarithmic transformation. We show that the discretized linear problem is dual to a Markov decision problem, the precise form of which depends on the chosen Galerkin basis. We prove a strong error bound in $L^{2}$ for the general scheme and discuss two special cases: a variant of the known Markov chain approximation obtained from a basis of characteristic functions of a box discretization, and a sparse approximation that uses the basis of committor functions of metastable sets of the dynamics; the latter is particularly suited for high-dimensional systems, e.g., control problems in molecular dynamics. We illustrate the method with several numerical examples, one being the optimal control of Alanine dipeptide to its helical conformation.

Motivation & Objective

• To develop a meshless discretization for LQ-type stochastic control problems that avoids traditional mesh-based PDE solvers.
• To address the computational inefficiency of PDE-based methods in high-dimensional systems.
• To establish a duality between the Galerkin discretization and Markov decision problems via the choice of basis functions.
• To provide a strong L² error bound for the general scheme, ensuring convergence.
• To enable efficient solution of high-dimensional control problems, particularly in molecular dynamics, using sparse bases like committor functions of metastable sets.

Proposed method

• The method employs a logarithmic transformation to convert the original nonlinear HJB equation into a linear boundary value problem.
• A Galerkin projection is applied using a user-defined basis of functions, transforming the continuous problem into a discrete linear system.
• The resulting discrete problem is shown to be dual to a Markov decision problem, with transition rates derived from the basis functions and the system dynamics.
• Two specific basis choices are analyzed: characteristic functions for box discretization (yielding a Markov chain approximation), and committor functions for sparse, metastability-aware discretization.
• The method ensures a strong L² error bound via best-approximation error control, proven in Appendix B.
• Sampling of the discretized running cost is enabled through milestoning and path probability formulas, as detailed in Appendix E.

Experimental results

Research questions

• RQ1 Can a meshless Galerkin method be effectively applied to LQ-type stochastic control problems with nonlinear state dependence?
• RQ2 How does the choice of Galerkin basis affect the dual Markov decision problem and the resulting discretization accuracy?
• RQ3 What is the convergence behavior of the method, and can a rigorous L² error bound be established?
• RQ4 Can the method efficiently handle high-dimensional systems, such as those in molecular dynamics?
• RQ5 How do committor function-based bases compare to standard Markov chain approximations in terms of sparsity and accuracy?

Key findings

• The proposed Galerkin scheme achieves a strong L² error bound, proven via best-approximation error control, ensuring convergence under appropriate basis selection.
• The discretized problem is dual to a Markov decision problem, with transition rates explicitly derived from the Galerkin basis and system parameters.
• Using characteristic functions of a box discretization recovers a known Markov chain approximation, validating the method’s consistency.
• The use of committor functions for metastable sets enables a sparse, high-dimensional discretization, particularly effective for systems with clear metastable states.
• Numerical results on a 1D triple-well potential and Alanine dipeptide show the method’s ability to compute optimal control forces for conformational transitions with high accuracy.
• The method enables efficient sampling of the discretized running cost via milestoning, as formalized in the sampling formula (34).