[Paper Review] Single Time-scale Actor-critic Method to Solve the Linear Quadratic Regulator with Convergence Guarantees

Mo Zhou, Jianfeng Lu|arXiv (Cornell University)|Jan 31, 2022

Frequency Control in Power Systems4 citations

TL;DR

This paper proposes a single timescale actor-critic algorithm for solving the linear quadratic regulator (LQR) problem using least squares temporal difference (LSTD) for the critic and natural policy gradient for the actor, achieving convergence with sample complexity 𝒪(ε⁻¹ log(ε⁻¹)²) to an ε-optimal solution, marking the first such provable guarantee in a single timescale framework for LQR.

ABSTRACT

We propose a single time-scale actor-critic algorithm to solve the linear quadratic regulator (LQR) problem. A least squares temporal difference (LSTD) method is applied to the critic and a natural policy gradient method is used for the actor. We give a proof of convergence with sample complexity $\mathcal{O}(\varepsilon^{-1} \log(\varepsilon^{-1})^2)$. The method in the proof is applicable to general single time-scale bilevel optimization problem. We also numerically validate our theoretical results on the convergence.

Motivation & Objective

• Address the challenge of convergence in single timescale actor-critic methods for optimal control, particularly in the absence of exact lower-level optimization.
• Overcome the error propagation issue common in bilevel optimization by simultaneously analyzing actor and critic updates.
• Provide the first convergence proof with explicit sample complexity for a single timescale actor-critic algorithm in the LQR setting.
• Develop a Lyapunov function that jointly tracks critic error and actor loss to establish contraction and convergence.
• Extend the method’s theoretical framework to general single timescale bilevel optimization problems beyond LQR.

Proposed method

• Apply least squares temporal difference (LSTD) to estimate the critic’s value function with explicit gradient computation and sample-based approximation.
• Use natural policy gradient updates for the actor to improve policy performance based on the critic’s value estimate.
• Construct a composite Lyapunov function combining critic estimation error and actor loss (J(Kₜ) − J(K*)) to jointly analyze convergence.
• Establish contraction of the Lyapunov function under two conditions: when either the critic error or actor loss is large, or both are small.
• Derive bounds on the update steps using spectral norms and matrix constants (e.g., σ_min(D_ε), c_D, c₃) to control error propagation.
• Use a constant step size βₜ and αₜ to ensure contraction rate (1 − βₜc₄) in the Lyapunov function, leading to convergence.

Experimental results

Research questions

• RQ1 Can a single timescale actor-critic method achieve convergence for the LQR problem without requiring two separate time scales?
• RQ2 What is the optimal sample complexity achievable by a single timescale actor-critic algorithm for LQR with provable guarantees?
• RQ3 How can the interplay between actor and critic errors be jointly analyzed to ensure convergence in the absence of exact lower-level optimization?
• RQ4 Can the proposed method be generalized to other single timescale bilevel optimization problems?
• RQ5 What role does the Lyapunov function play in ensuring contraction and stability in the joint actor-critic update process?

Key findings

• The proposed single timescale actor-critic method achieves convergence to an ε-optimal solution with sample complexity 𝒪(ε⁻¹ log(ε⁻¹)²), which is significantly better than existing two-timescale methods.
• The method proves convergence by constructing a Lyapunov function that contracts over time when either the critic error or actor loss is large, ensuring global convergence.
• The analysis does not require strong convexity of the higher-level loss, making it applicable to a broader class of problems than prior work.
• The Lyapunov function contraction is established via bounds on gradient differences and step size control, ensuring that the expected value of the function decreases at a rate (1 − βₜc₄).
• When both the critic error and actor loss fall below ε/2, the Lyapunov function remains below ε, indicating convergence to the optimal solution.
• The method achieves convergence without requiring exact solution of the critic’s problem at each step, enabling efficient and practical implementation.