[Paper Review] On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift

Alekh Agarwal, Sham M. Kakade+2 more|arXiv (Cornell University)|Aug 1, 2019

Reinforcement Learning in RoboticsComputer Science59 references111 citations

TL;DR

The paper provides global convergence guarantees for policy gradient methods in discounted MDPs, detailing both tabular and function-approximation settings, and introduces average-case approximation guarantees linked to distribution shift.

ABSTRACT

Policy gradient methods are among the most effective methods in challenging reinforcement learning problems with large state and/or action spaces. However, little is known about even their most basic theoretical convergence properties, including: if and how fast they converge to a globally optimal solution or how they cope with approximation error due to using a restricted class of parametric policies. This work provides provable characterizations of the computational, approximation, and sample size properties of policy gradient methods in the context of discounted Markov Decision Processes (MDPs). We focus on both: "tabular" policy parameterizations, where the optimal policy is contained in the class and where we show global convergence to the optimal policy; and parametric policy classes (considering both log-linear and neural policy classes), which may not contain the optimal policy and where we provide agnostic learning results. One central contribution of this work is in providing approximation guarantees that are average case -- which avoid explicit worst-case dependencies on the size of state space -- by making a formal connection to supervised learning under distribution shift. This characterization shows an important interplay between estimation error, approximation error, and exploration (as characterized through a precisely defined condition number).

Motivation & Objective

• Establish global convergence properties of policy gradient methods in tabular settings where the optimal policy lies in the class.
• Analyze convergence under function approximation when the optimal policy may lie outside the policy class.
• Provide average-case approximation guarantees that avoid worst-case state-space dependence via distribution shift concepts.
• Quantify the trade-offs between estimation error, approximation error, and exploration in policy optimization.

Proposed method

• Analyze first-order and quasi second-order policy gradient methods in discounted MDPs.
• Derive iteration complexity results for tabular policy gradient variants (projected gradient ascent on the simplex, softmax parameterization, and log-barrier regularized versions).
• Prove a fast convergence rate for Natural Policy Gradient (NPG) that is independent of state/action space size and D_infty under suitable conditions.
• Develop average-case guarantees for function-approximation policies using a transfer-error/distribution-shift framework.
• Utilize a policy-gradient formulation with policy parameters and the policy gradient theorem, including the gradient domination perspective.
• Provide comparisons to related approximate dynamic programming approaches through a structured error/decomposition analysis.

Experimental results

Research questions

• RQ1 Can policy gradient methods converge globally to the optimal policy in tabular settings when the optimal policy is within the policy class?
• RQ2 How do policy gradient methods perform under function approximation when the optimal policy may lie outside the chosen policy class?
• RQ3 What are the sample and iteration complexities for different policy gradient variants, and how do they depend on the discount factor and distributional properties?
• RQ4 How can average-case (distribution-shift) analysis yield practical, less pessimistic guarantees compared to worst-case bounds?
• RQ5 What role does regularization (such as log-barrier/entropy) play in achieving finite-time convergence for softmax parameterizations?

Key findings

• In the tabular setting, first-order methods achieve global convergence to the optimal policy under appropriate conditions.
• Natural Policy Gradient (NPG) achieves a convergence rate of 2/((1-γ)^2 ε), independent of state/action space size and distribution mismatch under the stated assumptions.
• Softmax parameterization with log-barrier regularization yields a finite-time iteration complexity bound that is polynomial in problem quantities.
• For function approximation, the paper provides average-case guarantees characterized by estimation error ε_stat and approximation error ε_approx, and a condition number κ, with convergence depending on distribution shift D_infty.
• The analysis connects policy gradient learning to supervised learning under distribution shift, enabling average-case guarantees that avoid explicit worst-case state-space dependencies.
• The results reveal a trade-off between estimation error, approximation error, and exploration (via the distribution mismatch coefficient D_infty) for first-order methods, and show dimension-free benefits for NPG.