[Paper Review] Solving Time-Continuous Stochastic Optimal Control Problems: Algorithm Design and Convergence Analysis of Actor-Critic Flow

Mo Zhou, Jianfeng Lu|arXiv (Cornell University)|Feb 27, 2024

Simulation Techniques and Applications4 citations

TL;DR

This paper proposes an actor-critic flow algorithm for solving time-continuous stochastic optimal control problems by combining least-squares temporal difference (LSTD) for value function estimation with policy gradient optimization. It establishes global linear convergence for the continuous gradient flow, validated numerically on benchmark problems.

ABSTRACT

We propose an actor-critic framework to solve the time-continuous stochastic optimal control problem. A least square temporal difference method is applied to compute the value function for the critic. The policy gradient method is implemented as policy improvement for the actor. Our key contribution lies in establishing a linear rate of convergence for our proposed actor-critic flow. Theoretical findings are further validated through numerical examples, showing the efficacy of our approach in practical applications.

Motivation & Objective

• Address the challenge of solving time-continuous stochastic optimal control problems with deterministic feedback control and without entropy regularization.
• Develop a machine learning-inspired framework that bridges reinforcement learning and optimal control theory.
• Ensure theoretical convergence guarantees for the proposed algorithm in the continuous-time limit.
• Design a critic component using least-squares temporal difference (LSTD) derived from Itô’s calculus to reduce error compared to standard RL TD methods.
• Achieve global linear convergence of the actor-critic flow under mild assumptions, providing a strong theoretical foundation for practical deployment.

Proposed method

• Formulate the stochastic optimal control problem using the Hamilton-Jacobi-Bellman (HJB) equation as the underlying PDE framework.
• Implement a critic component based on least-squares temporal difference (LSTD) estimation, derived from Itô’s lemma to improve accuracy over vanilla TD methods.
• Design the actor update via policy gradient method with an explicit gradient expression for policy improvement.
• Analyze the algorithm in the continuous limit, deriving the actor-critic flow as a continuous-time dynamical system.
• Establish global convergence using a Lyapunov-type argument, proving linear convergence rate under standard regularity and boundedness assumptions.
• Use a box-integration argument over compact sets to derive contradiction-based convergence proofs, leveraging bounds on state and control variables.

Experimental results

Research questions

• RQ1 Can an actor-critic framework be rigorously extended to time-continuous stochastic optimal control problems with deterministic policies?
• RQ2 Does the proposed LSTD-based critic yield better convergence and lower error than standard temporal difference methods in this continuous setting?
• RQ3 What is the convergence rate of the actor-critic flow in the continuous gradient limit, and can global convergence be guaranteed?
• RQ4 How does the policy gradient method with explicit gradient expression compare to vanilla gradient descent in terms of convergence behavior?
• RQ5 Can the theoretical convergence results be validated numerically on non-trivial stochastic control problems?

Key findings

• The proposed actor-critic flow achieves global linear convergence for the continuous gradient flow, a strong theoretical guarantee not commonly established in similar RL-based optimal control methods.
• The LSTD-based critic, derived from Itô’s calculus, reduces estimation error compared to standard temporal difference methods in the stochastic control context.
• The policy gradient method with explicit gradient expression outperforms vanilla gradient descent in convergence stability and speed.
• The convergence proof relies on a contradiction argument using box integration and bounds on state, control, and value function derivatives.
• Numerical examples validate the theoretical findings, demonstrating the algorithm’s efficacy in practical applications.
• The convergence rate is linear, and the proof holds under standard assumptions including Lipschitz continuity and boundedness of drift and diffusion coefficients.