[Paper Review] Learning Deep Stochastic Optimal Control Policies using Forward-Backward SDEs

Marcus A. Pereira, Ziyi Wang+2 more|arXiv (Cornell University)|Feb 11, 2019

Model Reduction and Neural Networks40 references25 citations

TL;DR

This paper proposes a deep learning framework for stochastic optimal control using forward-backward stochastic differential equations (FBSDEs), leveraging LSTM networks to model temporal dependencies and Girsanov's theorem for efficient exploration. The method achieves smoother, more scalable control policies than fully connected networks, especially under control constraints, with superior performance in pendulum, cart-pole, and quadcopter simulations.

ABSTRACT

In this paper we propose a new methodology for decision-making under uncertainty using recent advancements in the areas of nonlinear stochastic optimal control theory, applied mathematics, and machine learning. Grounded on the fundamental relation between certain nonlinear partial differential equations and forward-backward stochastic differential equations, we develop a control framework that is scalable and applicable to general classes of stochastic systems and decision-making problem formulations in robotics and autonomy. The proposed deep neural network architectures for stochastic control consist of recurrent and fully connected layers. The performance and scalability of the aforementioned algorithm are investigated in three non-linear systems in simulation with and without control constraints. We conclude with a discussion on future directions and their implications to robotics.

Motivation & Objective

• To develop a scalable, differentiable framework for stochastic optimal control in nonlinear systems under uncertainty.
• To overcome the curse of dimensionality and limitations of traditional dynamic programming in high-dimensional stochastic control.
• To improve control policy smoothness and training efficiency by replacing fully connected networks with recurrent LSTMs in FBSDE-based control.
• To extend the framework to handle control constraints, which are critical in real-world robotics applications.
• To demonstrate scalability and performance on complex systems like pendulums, cart-poles, and quadcopters with realistic actuator limits.

Proposed method

• Uses forward-backward stochastic differential equations (FBSDEs) to reformulate the Hamilton-Jacobi-Bellman (HJB) equation for stochastic optimal control.
• Applies Girsanov's theorem to reweight trajectories and enable efficient exploration by modifying drift terms in the forward SDE.
• Employs Long-Short Term Memory (LSTM) networks to learn temporal dependencies in control policies, replacing independent fully connected networks at each time step.
• Uses a deep learning-based algorithm to approximate the solution of the backward SDE via neural network estimation of conditional expectations.
• Implements a differentiable architecture that jointly optimizes the control policy and value function using end-to-end backpropagation.
• Extends the framework to handle control constraints by incorporating box constraints on control inputs during policy learning.

Experimental results

Research questions

• RQ1 Can FBSDE-based stochastic optimal control be effectively learned using deep neural networks with recurrent architectures?
• RQ2 How does using LSTMs instead of fully connected layers improve control policy smoothness, training efficiency, and scalability in stochastic control?
• RQ3 To what extent can Girsanov's theorem enhance exploration in FBSDE-based control without relying on restrictive assumptions between control and noise?
• RQ4 How does the proposed method perform on high-dimensional, nonlinear systems with control constraints compared to fully connected networks?
• RQ5 Can the framework be extended to systems with non-affine control dynamics or Lévy-type noise?

Key findings

• The LSTM-based policy reduced training time by at least 20% and used significantly fewer parameters than the fully connected network across all tasks.
• Control policies learned with LSTMs produced smoother control trajectories, avoiding the spiky, sawtooth-like controls observed in fully connected networks under constraints.
• The LSTM policy achieved successful swing-up in pendulum and cart-pole tasks with lower variance and better handling of momentum, especially in unconstrained settings.
• For the quadcopter task with a 2-second horizon, the FC-based policy required 100 separate networks and proved difficult to tune, while the LSTM-based policy scaled efficiently with shared weights and faster graph construction.
• The LSTM policy outperformed the FC policy in tracking target positions and attitudes in the quadcopter task, particularly in the Z and yaw states, due to better long-term temporal modeling.
• The framework successfully handled control constraints in all three systems, with LSTM-based policies maintaining smooth control inputs even under torque limits, unlike the FC counterparts.