[Paper Review] Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization

Shicong Cen, Cheng Chen+3 more|arXiv (Cornell University)|Jul 13, 2020

Reinforcement Learning in Robotics58 references41 citations

TL;DR

This paper proves non-asymptotic, linear (and locally quadratic) convergence guarantees for entropy-regularized natural policy gradient methods with softmax parameterization in discounted MDPs, including stability under approximate policy evaluation.

ABSTRACT

Natural policy gradient (NPG) methods are among the most widely used policy optimization algorithms in contemporary reinforcement learning. This class of methods is often applied in conjunction with entropy regularization -- an algorithmic scheme that encourages exploration -- and is closely related to soft policy iteration and trust region policy optimization. Despite the empirical success, the theoretical underpinnings for NPG methods remain limited even for the tabular setting. This paper develops $ extit{non-asymptotic}$ convergence guarantees for entropy-regularized NPG methods under softmax parameterization, focusing on discounted Markov decision processes (MDPs). Assuming access to exact policy evaluation, we demonstrate that the algorithm converges linearly -- or even quadratically once it enters a local region around the optimal policy -- when computing optimal value functions of the regularized MDP. Moreover, the algorithm is provably stable vis-à-vis inexactness of policy evaluation. Our convergence results accommodate a wide range of learning rates, and shed light upon the role of entropy regularization in enabling fast convergence.

Motivation & Objective

• Motivate and understand the theoretical performance of entropy-regularized natural policy gradient (NPG) methods in tabular, discounted MDPs.
• Establish non-asymptotic convergence rates for exact and approximate policy evaluation.
• Characterize how entropy regularization influences convergence speed and stability across learning rates.
• Show potential for fast convergence and provide comparisons to related policy optimization methods.

Proposed method

• Study entropy-regularized NPG with softmax parameterization under a γ-discounted MDP.
• Derive an exact update rule in policy space showing a closed-form connection to softmax with Qτ under regularization.
• Prove non-asymptotic linear convergence for exact policy evaluation with general learning rate η in (0,(1−γ)/τ].
• Extend to approximate policy evaluation, proving stability and an error floor proportional to the Q-function estimation error.
• Demonstrate quadratic convergence in a small-ε regime when η = (1−γ)/τ, leading to log-log(1/ε) iteration scaling after entering a local region.
• Discuss comparisons to TRPO, SPI, and related methods and provide a bandit warm-up illustrating linear convergence.

Experimental results

Research questions

• RQ1 Can entropy regularization accelerate convergence of natural policy gradient methods in tabular discounted MDPs under softmax parameterization?
• RQ2 What non-asymptotic convergence rates (linear, quadratic) can be guaranteed for exact and approximate entropy-regularized NPG methods?
• RQ3 How do learning rate and regularization parameter interact to affect convergence speed and stability when policy evaluation is exact or approximate?
• RQ4 How does entropy regularization relate to and improve upon existing methods like SPI and TRPO in terms of iteration complexity?
• RQ5 What is the impact of approximate Q-function accuracy on overall convergence and error floors?

Key findings

• Entropy-regularized NPG converges linearly to the optimal soft Q-function and the log-policy when policy evaluation is exact.
• For any η in (0,(1−γ)/τ], the iteration to achieve ε-optimality in the regularized MDP scales as (1/(ητ)) log(1/ε).
• Approximate entropy-regularized NPG remains linearly convergent up to an error floor that grows linearly with the entrywise error of the soft Q-function estimates.
• In the small-ε regime, the update with η = (1−γ)/τ achieves quadratic (super-linear) convergence, yielding iteration complexity on the order of log log(1/ε) after entering a local region.
• Compared to unregularized NPG (or PG), entropy regularization enables faster convergence, with a tabular complexity gap highlighted relative to prior work.
• The bandit warm-up demonstrates linear convergence of the entropy-regularized NPG update to the soft-max optimal policy, with potential single-shot convergence when η = 1/τ.