[Paper Review] An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods

Yanli Liu, Kaiqing Zhang+2 more|arXiv (Cornell University)|Nov 15, 2022

Stochastic Gradient Optimization Techniques36 references30 citations

TL;DR

The paper revisits and strengthens global convergence analyses for policy gradient (PG), natural PG (NPG), and their variance-reduced variants, showing global convergence up to function-approximation error and improved sample complexities, including a new SRVR-NPG method.

ABSTRACT

In this paper, we revisit and improve the convergence of policy gradient (PG), natural PG (NPG) methods, and their variance-reduced variants, under general smooth policy parametrizations. More specifically, with the Fisher information matrix of the policy being positive definite: i) we show that a state-of-the-art variance-reduced PG method, which has only been shown to converge to stationary points, converges to the globally optimal value up to some inherent function approximation error due to policy parametrization; ii) we show that NPG enjoys a lower sample complexity; iii) we propose SRVR-NPG, which incorporates variance-reduction into the NPG update. Our improvements follow from an observation that the convergence of (variance-reduced) PG and NPG methods can improve each other: the stationary convergence analysis of PG can be applied to NPG as well, and the global convergence analysis of NPG can help to establish the global convergence of (variance-reduced) PG methods. Our analysis carefully integrates the advantages of these two lines of works. Thanks to this improvement, we have also made variance-reduction for NPG possible, with both global convergence and an efficient finite-sample complexity.

Motivation & Objective

• Motivate and establish global convergence guarantees for PG and NPG under general smooth policy parametrizations.
• Improve existing global convergence rates from prior work for NPG and VR-PG methods.
• Introduce SRVR-NPG to incorporate variance reduction into natural policy gradient.
• Demonstrate global convergence of SRVR-PG and SRVR-NPG with finite-sample guarantees.
• Provide theoretical guidance for sample complexities and function-approximation biases in practical RL settings.

Proposed method

• Develop a general convergence framework linking stationary convergence and update-direction accuracy to global policy performance.
• Assume positive definiteness of the Fisher information matrix to enable preconditioned updates and relate to existing NPG theory.
• Apply variance-reduction to PG and NPG, yielding SRVR-PG and SRVR-NPG, with analysis of their global convergence.
• Derive non-asymptotic sample complexity results for PG, NPG, SRVR-PG, and SRVR-NPG under standard RL assumptions.
• Incorporate truncated GPOMDP estimators and importance-weighted corrections to enable finite-sample analysis.

Experimental results

Research questions

• RQ1 Can variance-reduced PG methods (SRVR-PG) achieve global convergence to near-optimal policies under function approximation errors?
• RQ2 Does natural policy gradient (NPG) admit improved global convergence rates and sample complexity when combined with variance reduction (SRVR-NPG)?
• RQ3 How does the positive definiteness of the Fisher information matrix influence the convergence and sample complexity of PG/NPG methods?
• RQ4 What are the finite-sample requirements (trajectories, horizon, iterations) to guarantee policy performance within a bias term of the optimum?

Key findings

• SRVR-PG converges globally up to a function-approximation error with a sample complexity of O(epsilon^{-3}).
• NPG achieves improved global convergence with O(epsilon^{-3}) sample complexity or better under the proposed framework, improving upon prior O(epsilon^{-4}) results.
• SRVR-NPG extends variance reduction to NPG, achieving global convergence with finite-sample guarantees comparable to improved NPG results.
• Assuming positive definiteness of the Fisher information matrix, the analyses show that stationary convergence and global convergence analyses can inform each other for both PG and NPG.
• The paper proves that variance reduction can be integrated into NPG, enabling efficient, global convergence with practical sample complexity.
• Numerical experiments on Cartpole and Mountain Car indicate SRVR-NPG provides the best empirical performance among the methods tested.