[Paper Review] Stochastic Variance-Reduced Policy Gradient

Matteo Papini, Damiano Binaghi+3 more|arXiv (Cornell University)|Jun 14, 2018

Reinforcement Learning in Robotics16 citations

TL;DR

This paper introduces SVRPG, a stochastic variance-reduced policy gradient algorithm for reinforcement learning that addresses non-concave objectives, approximate full gradients, and non-stationary sampling in MDPs. By leveraging importance weights to preserve gradient unbiasedness, SVRPG achieves linear convergence under increasing batch sizes, demonstrating improved sample efficiency and stability on continuous control tasks.

ABSTRACT

In this paper, we propose a novel reinforcement- learning algorithm consisting in a stochastic variance-reduced version of policy gradient for solving Markov Decision Processes (MDPs). Stochastic variance-reduced gradient (SVRG) methods have proven to be very successful in supervised learning. However, their adaptation to policy gradient is not straightforward and needs to account for I) a non-concave objective func- tion; II) approximations in the full gradient com- putation; and III) a non-stationary sampling pro- cess. The result is SVRPG, a stochastic variance- reduced policy gradient algorithm that leverages on importance weights to preserve the unbiased- ness of the gradient estimate. Under standard as- sumptions on the MDP, we provide convergence guarantees for SVRPG with a convergence rate that is linear under increasing batch sizes. Finally, we suggest practical variants of SVRPG, and we empirically evaluate them on continuous MDPs.

Motivation & Objective

• To develop a variance-reduced policy gradient method suitable for reinforcement learning, where standard SVRG techniques face challenges due to non-concave objectives and non-stationary data distributions.
• To address the three key difficulties in adapting SVRG to policy gradient: non-concave objective functions, approximations in full gradient computation, and non-stationary sampling processes.
• To maintain unbiased gradient estimates in the presence of importance weighting while ensuring convergence under standard MDP assumptions.
• To provide theoretical convergence guarantees with a linear rate under increasing batch sizes, improving upon standard stochastic policy gradient methods.
• To empirically validate the method on continuous control benchmarks, showing improved sample efficiency and stability compared to baseline policy gradient approaches.

Proposed method

• Proposes SVRPG, a novel policy gradient algorithm that combines the SVRG framework with policy gradient by using two distinct ADAM optimizers: one for snapshot updates and one for sub-iterations.
• Employs importance weighting to correct for distributional shifts between the current policy and the snapshot policy, preserving the unbiasedness of the gradient estimate.
• Uses a two-level optimization loop: outer loop computes a full gradient estimate at a snapshot policy, and inner loop performs multiple stochastic updates using mini-batches with variance reduction.
• Applies separate learning rates and adaptive moment histories for the snapshot and sub-iteration phases, with the snapshot ADAM using a larger batch size (N) and a higher learning rate.
• Introduces a practical variant using a linear baseline (critic) for variance reduction in the Half Cheetah task, fitted via least squares at each snapshot.
• Employs a finite-sum approximation of the expected policy gradient, where the full gradient is estimated using a large batch of trajectories, and the stochastic gradient is computed via mini-batches.

Experimental results

Research questions

• RQ1 Can variance-reduced gradient methods like SVRG be successfully adapted to policy gradient reinforcement learning despite non-concave objectives and non-stationary sampling?
• RQ2 How can unbiased gradient estimation be preserved in policy gradient when using importance weighting and approximate full gradients?
• RQ3 What convergence rate can be theoretically guaranteed for a variance-reduced policy gradient algorithm under standard MDP assumptions?
• RQ4 How does SVRPG compare in sample efficiency and stability to standard policy gradient baselines on continuous control tasks?
• RQ5 What practical configurations of meta-parameters and adaptive optimizers yield the best performance in real-world RL environments?

Key findings

• SVRPG achieves a linear convergence rate of O(1/T) under standard MDP assumptions, with convergence improving as batch sizes increase.
• The algorithm demonstrates superior sample efficiency and stability on continuous control tasks such as Half Cheetah, Swimmer, and Cart-Pole, outperforming standard policy gradient baselines.
• Empirical results show that using separate ADAM optimizers for snapshots and sub-iterations significantly improves training stability and convergence speed.
• The inclusion of a linear baseline in Half Cheetah reduces variance and accelerates learning, with the critic updated only at snapshot points.
• SVRPG maintains consistent performance across different environments, with minimal hyperparameter tuning, particularly when using importance weighting to correct for distributional shift.
• The method achieves strong performance on Mujoco benchmarks, with agents successfully learning to balance and move forward in complex, high-dimensional state-action spaces.