[Paper Review] A Finite Time Analysis of Two Time-Scale Actor Critic Methods

Yue Wu, Weitong Zhang+2 more|arXiv (Cornell University)|May 4, 2020

Reinforcement Learning in RoboticsComputer Science38 references37 citations

TL;DR

The paper provides the first non-asymptotic analysis of two time-scale actor-critic methods with Markovian samples, proving convergence to an approximate stationary point and giving a a0icient a0sample complexity of e(epsilon^{-2.5}) for finding an
b0-stationary point.

ABSTRACT

Actor-critic (AC) methods have exhibited great empirical success compared with other reinforcement learning algorithms, where the actor uses the policy gradient to improve the learning policy and the critic uses temporal difference learning to estimate the policy gradient. Under the two time-scale learning rate schedule, the asymptotic convergence of AC has been well studied in the literature. However, the non-asymptotic convergence and finite sample complexity of actor-critic methods are largely open. In this work, we provide a non-asymptotic analysis for two time-scale actor-critic methods under non-i.i.d. setting. We prove that the actor-critic method is guaranteed to find a first-order stationary point (i.e., $\| abla J(\boldsymbolθ)\|_2^2 \le ε$) of the non-concave performance function $J(\boldsymbolθ)$, with $\mathcal{ ilde{O}}(ε^{-2.5})$ sample complexity. To the best of our knowledge, this is the first work providing finite-time analysis and sample complexity bound for two time-scale actor-critic methods.

Motivation & Objective

• Motivate the study of finite-time convergence for two time-scale actor-critic (AC) algorithms under non-i.i.d. data.
• Provide non-asymptotic convergence guarantees for an on-line, one-step AC method with linear TD(0) critic.
• Characterize the interplay between actor and critic updates under Markovian noise.
• Derive the sample complexity and rate to reach a first-order stationary point.
• Highlight how the proposed analysis improves understanding over decoupled or i.i.d.-assuming setups.

Proposed method

• Analyze the classical two time-scale actor-critic algorithm with TD(0) critic and linear function approximation.
• Assume bounded feature norm and establish TD(0) limiting point (c1) with matrix  and vector .
• Prove actor convergence under non-i.i.d. Markovian samples with step sizes ant t and et a t satisfying 0<c1<1, 0<nu<cs<1.
• Show Lipschitz continuity of the critic solution (c1) with respect to the policy parameter via Assumptions 4.1-4.3 and Proposition 4.4.
• Derive the overall convergence rate  in terms of pp (approximation error) and an optimization error term, yielding  = (pp) + O(t^{-(1-c})) + O(( log t)/t^{2}) + O((t)).
• Conclude that the total sample complexity to obtain an ps-stationary point is  = (pp) + e(
  b5^{-2.5}) under chosen s and t.

Experimental results

Research questions

• RQ1 Can two time-scale actor-critic methods achieve non-asymptotic convergence under non-i.i.d. (Markovian) samples with linear function approximation?
• RQ2 What is the finite-sample complexity to reach an ps-stationary point of the non-concave performance function J(m btheta)?
• RQ3 How do the actor and critic step-sizes influence the convergence rate and overall sample complexity?
• RQ4 How does the analysis compare to decoupled actor-critic and i.i.d.-assumption results?
• RQ5 Does the framework extend to alternative policy evaluation schemes and non-linear approximators?

Key findings

• The actor-critic method converges to an ps-approximate stationary point of J with (ps) = (pp) + O(t^{-(1-c)}) + O((log t)/t^{}) + O((t)).
• With s = O(1/t^{3/5}) for the actor and t = O(1/t^{2/5}) for the critic, the method attains ps-stationarity in T = (ps) iterations; the per-iteration sample is 1.
• The overall (finite-time) sample complexity is  = (pp) + ilde{O}(ps^{-2.5}).
• The analysis handles Markovian noise and removes the need for i.i.d. data assumptions, unlike some prior works.
• The authors propose a new proof framework that tightly bounds critic-estimation error and avoids extra artificial factors present in some iterative refinement approaches.
• Compared to decoupled actor-critic methods, the two time-scale approach is more sample-efficient, achieving  = ilde{O}(ps^{-2.5}) vs.  = ilde{O}(ps^{-4}) in some decoupled analyses.