[Paper Review] Variance-Reduced Off-Policy Memory-Efficient Policy Search

Daoming Lyu, Qi Qi+4 more|arXiv (Cornell University)|Sep 14, 2020

Reinforcement Learning in Robotics43 references4 citations

TL;DR

This paper proposes VOMPS and ACE-STORM, the first variance-reduced, off-policy, memory-efficient policy gradient algorithms that combine stochastic recursive momentum (STORM) with emphatic-weighted actor-critic frameworks (ACE/GeoffPAC). By eliminating the need for large reference sample sets and enabling stable off-policy learning, the methods achieve faster convergence and lower variance than prior on-policy or memory-heavy approaches, with empirical results showing superior sample efficiency and robustness to action noise.

ABSTRACT

Off-policy policy optimization is a challenging problem in reinforcement learning (RL). The algorithms designed for this problem often suffer from high variance in their estimators, which results in poor sample efficiency, and have issues with convergence. A few variance-reduced on-policy policy gradient algorithms have been recently proposed that use methods from stochastic optimization to reduce the variance of the gradient estimate in the REINFORCE algorithm. However, these algorithms are not designed for the off-policy setting and are memory-inefficient, since they need to collect and store a large ``reference'' batch of samples from time to time. To achieve variance-reduced off-policy-stable policy optimization, we propose an algorithm family that is memory-efficient, stochastically variance-reduced, and capable of learning from off-policy samples. Empirical studies validate the effectiveness of the proposed approaches.

Motivation & Objective

• Address the dual challenges of high variance and memory inefficiency in off-policy policy optimization.
• Enable stable, variance-reduced policy learning from off-policy data without storing large reference batches.
• Develop the first off-policy variance-reduced policy gradient methods that are both convergent and memory-efficient.
• Integrate STORM’s recursive momentum with off-policy actor-critic frameworks (ACE/GeoffPAC) to achieve stable, low-variance updates.
• Validate the proposed algorithms on benchmark control environments with improved sample efficiency and noise resistance.

Proposed method

• Leverage STORM, a stochastic variance-reduction technique that avoids fixed reference batches by using recursive momentum, to reduce gradient variance without storing large sample sets.
• Integrate STORM with the ACE and GeoffPAC off-policy actor-critic frameworks to ensure stability in off-policy learning.
• Use emphatic weighting in ACE and GeoffPAC to correct for distributional shift between behavior and target policies, enhancing off-policy convergence.
• Design VOMPS as a variance-reduced off-policy policy gradient method and ACE-STORM as a variance-reduced actor-critic variant, both avoiding reference sample storage.
• Apply recursive gradient updates that combine current stochastic gradients with momentum from past iterates, enabling variance reduction without reprocessing old data.
• Use importance sampling and off-policy correction in the gradient estimation to maintain stability when learning from behavior policy data.

Experimental results

Research questions

• RQ1 Can stochastic variance reduction be effectively applied to off-policy policy learning without requiring large memory for reference samples?
• RQ2 Can STORM’s recursive momentum mechanism be integrated with off-policy actor-critic frameworks to achieve both variance reduction and memory efficiency?
• RQ3 How does the proposed method compare in sample efficiency and stability to existing on-policy variance-reduced methods and off-policy baselines?
• RQ4 Does the proposed algorithm maintain robustness under action noise and distributional shift in off-policy settings?
• RQ5 Can the combination of STORM and emphatic weighting enable stable, convergent, and low-variance policy optimization in continuous control tasks?

Key findings

• VOMPS and ACE-STORM achieve near-optimal performance on CartPoleContinuous-v0 with around 200K samples, outperforming SVRPG and SRVR-PG, which require over 400K samples.
• On Hopper-v2 and HalfCheetah-v2, VOMPS and ACE-STORM converge faster and achieve higher Monte Carlo returns than SVRPG, SRVR-PG, DDPG, and TD3, with significantly lower variance.
• The learning curves of VOMPS and ACE-STORM are smoother and more stable than those of GeoffPAC, DDPG, and TD3, indicating better convergence properties.
• Under 20% action noise, VOMPS and ACE-STORM show superior robustness, remaining stable and effective, while other methods degrade significantly.
• ACE-STORM and VOMPS demonstrate the highest noise resistance among all methods tested, confirming the effectiveness of stochastic variance reduction in real-world perturbations.
• The proposed algorithms are the first to simultaneously achieve off-policy stability, variance reduction, and memory efficiency, outperforming existing on-policy variance-reduced methods in sample efficiency and stability.