[論文レビュー] A Finite Time Analysis of Two Time-Scale Actor Critic Methods

Yue Wu, Weitong Zhang+2 more|arXiv (Cornell University)|May 4, 2020

Reinforcement Learning in Robotics参考文献 38被引用数 37

ひとこと要約

tldr: 本論文は、Markovian samples を用いた二時相 actor-critic 法の最初の非漸近解析を提供し、近似的な stationary point への収束を証明し、ある 0 -stationary point を見つけるための 0sample complexity を与える。

ABSTRACT

Actor-critic (AC) methods have exhibited great empirical success compared with other reinforcement learning algorithms, where the actor uses the policy gradient to improve the learning policy and the critic uses temporal difference learning to estimate the policy gradient. Under the two time-scale learning rate schedule, the asymptotic convergence of AC has been well studied in the literature. However, the non-asymptotic convergence and finite sample complexity of actor-critic methods are largely open. In this work, we provide a non-asymptotic analysis for two time-scale actor-critic methods under non-i.i.d. setting. We prove that the actor-critic method is guaranteed to find a first-order stationary point (i.e., $\| abla J(\boldsymbolθ)\|_2^2 \le ε$) of the non-concave performance function $J(\boldsymbolθ)$, with $\mathcal{ ilde{O}}(ε^{-2.5})$ sample complexity. To the best of our knowledge, this is the first work providing finite-time analysis and sample complexity bound for two time-scale actor-critic methods.

研究の動機と目的

• Motivate the study of finite-time convergence for two time-scale actor-critic (AC) algorithms under non-i.i.d. data.
• Provide non-asymptotic convergence guarantees for an on-line, one-step AC method with linear TD(0) critic.
• Characterize the interplay between actor and critic updates under Markovian noise.
• Derive the sample complexity and rate to reach a first-order stationary point.
• Highlight how the proposed analysis improves understanding over decoupled or i.i.d.-assuming setups.

提案手法

• Analyze the classical two time-scale actor-critic algorithm with TD(0) critic and linear function approximation.
• Assume bounded feature norm and establish TD(0) limiting point \u001f(\u0003c1) with matrix \u001d and vector \u001b.
• Prove actor convergence under non-i.i.d. Markovian samples with step sizes \u001cant t and \u001bet a t satisfying 0<\u0003c1<1, 0<\u0003nu<\u0003cs<1.
• Show Lipschitz continuity of the critic solution \u001f(\u0003c1) with respect to the policy parameter via Assumptions 4.1-4.3 and Proposition 4.4.
• Derive the overall convergence rate \u001f in terms of \u001app (approximation error) and an optimization error term, yielding \u001f = \u001f(\u001app) + O(t^{-(1-\u0003c)}) + O((\u001c log t)/t^{\u001c2}) + O(\u001e(t)).
• Conclude that the total sample complexity to obtain an \u001eps-stationary point is \u001f = \u001f(\u001app) + \u0007e(\u001b5^{-2.5}) under chosen \u001cs and \u001dt.

実験結果

リサーチクエスチョン

• RQ1 Can two time-scale actor-critic methods achieve non-asymptotic convergence under non-i.i.d. (Markovian) samples with linear function approximation?
• RQ2 What is the finite-sample complexity to reach an \u001eps-stationary point of the non-concave performance function J(\u001bm \u0003btheta)?
• RQ3 How do the actor and critic step-sizes influence the convergence rate and overall sample complexity?
• RQ4 How does the analysis compare to decoupled actor-critic and i.i.d.-assumption results?
• RQ5 Does the framework extend to alternative policy evaluation schemes and non-linear approximators?

主な発見

• The actor-critic method converges to an \u001eps-approximate stationary point of J with 0(\u001eps) = 0(\u001app) + O(t^{-(1-\u0003c)}) + O((log t)/t^{\u001c}) + O(\u001e(t)).
• With \u001cs = O(1/t^{3/5}) for the actor and \u001dt = O(1/t^{2/5}) for the critic, the method attains \u001eps-stationarity in T = 0(\u001eps) iterations; the per-iteration sample is 1.
• The overall (finite-time) sample complexity is 0 = 0(\u001app) + \\tilde{O}(\u001eps^{-2.5}).
• The analysis handles Markovian noise and removes the need for i.i.d. data assumptions, unlike some prior works.
• The authors propose a new proof framework that tightly bounds critic-estimation error and avoids extra artificial factors present in some iterative refinement approaches.
• Compared to decoupled actor-critic methods, the two time-scale approach is more sample-efficient, achieving 0 = \\tilde{O}(\u001eps^{-2.5}) vs. 0 = \\tilde{O}(\u001eps^{-4}) in some decoupled analyses.