[Paper Review] A deep learning method for solving stochastic optimal control problems driven by fully-coupled FBSDEs

Shaolin Ji, Shigē Péng+2 more|arXiv (Cornell University)|Apr 12, 2022

Energy, Environment, and Transportation Policies4 citations

TL;DR

This paper proposes a deep learning method based on a cross-optimization (CO) framework to solve high-dimensional stochastic optimal control problems driven by fully-coupled forward-backward stochastic differential equations (FBSDEs). By reformulating the problem as a stochastic Stackelberg differential game, the method alternately optimizes the leader’s control and the follower’s state variables using deep neural networks, achieving accurate solutions in two investment-consumption examples with recursive utility models.

ABSTRACT

In this paper,we mainly focus on the numerical solution of high-dimensional stochastic optimal control problem driven by fully-coupled forward-backward stochastic differential equations (FBSDEs in short) through deep learning. We first transform the problem into a stochastic Stackelberg differential game problem (leader-follower problem), then a bi-level optimization method is developed where the leader's cost functional and the follower's cost functional are optimized alternatively via deep neural networks. As for the numerical results, we compute two examples of the investment-consumption problem solved through stochastic recursive utility models, and the results of both examples demonstrate the effectiveness of our proposed algorithm.

Motivation & Objective

• To address the lack of numerical methods for high-dimensional stochastic optimal control problems governed by fully-coupled FBSDEs.
• To overcome the computational difficulty of solving high-dimensional FBSDEs using traditional PDE or probabilistic methods.
• To develop a scalable deep learning framework that can handle the coupled forward and backward dynamics in FBSDEs.
• To apply the method to stochastic recursive utility models in portfolio optimization, demonstrating its effectiveness in financial applications.

Proposed method

• Reformulate the stochastic optimal control problem as a stochastic Stackelberg differential game, where the leader controls the system and the follower determines the state and adjoint processes.
• Construct three feed-forward neural networks: one to approximate the leader’s control $u(\cdot)$, and two to approximate the initial value $y_0$ and the diffusion coefficient $z(\cdot)$ of the backward component.
• Define two cost functionals: $J(u(\cdot))$ for the leader’s optimization and $J(y_0, z(\cdot))$ for the follower’s minimization of terminal condition error.
• Implement a cross-optimization (CO) algorithm that alternately updates the neural network parameters: perform $\kappa \geq 1$ steps for the follower’s network before one update for the leader’s network.
• Use Monte Carlo sampling to estimate the expected values in the cost functionals, enabling stochastic gradient descent training.
• Apply the method to two investment-consumption problems with recursive utility, using numerical experiments to validate performance.

Experimental results

Research questions

• RQ1 Can deep learning effectively solve high-dimensional stochastic optimal control problems driven by fully-coupled FBSDEs?
• RQ2 How does the proposed Stackelberg-based cross-optimization method compare to existing numerical schemes for FBSDEs in high dimensions?
• RQ3 What is the impact of the number of follower updates ($\kappa$) on the convergence and stability of the solution?
• RQ4 To what extent can the method accurately recover the initial value $y_0$ and control $u(\cdot)$ in recursive utility models?
• RQ5 How robust is the method across different time horizons and dimensions in the investment-consumption problem?

Key findings

• The method successfully solves two high-dimensional investment-consumption problems under recursive utility, achieving stable and accurate results across multiple time horizons.
• For $T=1.00$ and $n=50$, the mean distance between the integral and parametric forms of $y_0$ was $0.00101 \pm 1.36 \times 10^{-7}$, indicating high convergence accuracy.
• When $\kappa=1$, the distance between the two $y_0$ forms was $0.00124 \pm 1.58 \times 10^{-7}$, but this increased significantly to $0.06069 \pm 1.11 \times 10^{-7}$ at $\kappa=1/49$, showing that $\kappa>1$ improves stability.
• With $\kappa=9$, the distance dropped to $0.00043 \pm 2.08 \times 10^{-7}$, demonstrating that more follower updates lead to better alignment between solution forms.
• The parametric and integral forms of $y_0$ converged closely over iterations, with the mean value stabilizing after 5000 training steps in the $n=50$, $T=0.5$ case.
• The results confirm that the CO method is effective and robust, especially when $\kappa > 1$, and outperforms baseline settings with $\kappa=1$.