[Paper Review] Off-Policy Actor-Critic with Shared Experience Replay

Simon Schmitt, Matteo Hessel+1 more|arXiv (Cornell University)|Sep 25, 2019

Reinforcement Learning in Robotics39 references4 citations

TL;DR

This paper proposes LASER, an off-policy actor-critic reinforcement learning agent that uses shared, large-scale experience replay to achieve state-of-the-art data efficiency. By mixing on-policy experience with replayed experience and employing a trust region scheme, LASER enables stable, high-sample-efficiency learning in strongly off-policy settings, achieving 448% human-normalized score on 57 Atari games at 200M environment steps.

ABSTRACT

We investigate the combination of actor-critic reinforcement learning algorithms with uniform large-scale experience replay and propose solutions for two challenges: (a) efficient actor-critic learning with experience replay (b) stability of off-policy learning where agents learn from other agents behaviour. We employ those insights to accelerate hyper-parameter sweeps in which all participating agents run concurrently and share their experience via a common replay module. To this end we analyze the bias-variance tradeoffs in V-trace, a form of importance sampling for actor-critic methods. Based on our analysis, we then argue for mixing experience sampled from replay with on-policy experience, and propose a new trust region scheme that scales effectively to data distributions where V-trace becomes unstable. We provide extensive empirical validation of the proposed solution. We further show the benefits of this setup by demonstrating state-of-the-art data efficiency on Atari among agents trained up until 200M environment frames.

Motivation & Objective

• Address the instability and lack of convergence in off-policy actor-critic learning when using experience replay.
• Improve data efficiency in deep reinforcement learning by enabling concurrent agents to share a common experience replay buffer.
• Overcome the bias-variance trade-off in V-trace by analyzing its limitations in off-policy settings.
• Develop a trust region scheme that ensures policy improvement even under strong off-policy sampling.
• Demonstrate that mixing on-policy experience with replayed experience is essential for convergence and performance.

Proposed method

• Propose a hybrid training strategy that combines on-policy experience with off-policy experience from a shared replay buffer to reduce bias in V-trace.
• Introduce a trust region scheme in policy space that guarantees policy improvement by constraining updates to maintain stability.
• Use V-trace with importance sampling for off-policy correction, but mitigate its bias through on-policy mixing and trust region constraints.
• Implement a shared replay buffer across multiple agents to enable parallel exploration and reduce memory footprint.
• Apply uniform experience replay instead of prioritized replay to avoid agent-specific metric conflicts in shared settings.
• Train agents using population-based training (PBT) with shared experience, enabling efficient hyper-parameter sweeps.

Experimental results

Research questions

• RQ1 Can off-policy actor-critic learning with experience replay converge to a locally optimal policy without on-policy experience?
• RQ2 How does mixing on-policy and off-policy experience affect the stability and performance of V-trace in off-policy actor-critic methods?
• RQ3 What is the impact of shared experience replay on data efficiency and training speed in concurrent hyper-parameter sweeps?
• RQ4 Can a trust region scheme stabilize learning in strongly off-policy regimes where V-trace becomes unstable?
• RQ5 Does uniform experience replay outperform prioritized replay in multi-agent shared replay settings?

Key findings

• Mixing on-policy experience with replayed experience significantly improves performance and ensures convergence, as pure off-policy V-trace fails to converge even with optimal value functions.
• LASER with shared experience replay achieves 448% human-normalized score on 57 Atari games at 200M environment steps, outperforming prior methods that required over 3B steps to exceed 400%.
• The best performance is achieved with a 10M replay buffer and 87.5% replay ratio (7:1 replay-to-online ratio), indicating high data efficiency with moderate replay capacity.
• Shared experience replay reduces memory usage by 10× in 10-agent sweeps (from 100M to 10M) while enabling parallel exploration and faster hyper-parameter sweeps.
• The trust region scheme enables stable learning in strongly off-policy settings, outperforming non-trust-region baselines even when using shared experience.
• LASER achieves 97.2% median human-normalized performance on DMLab-30 at 10B steps, demonstrating strong multi-task generalization and efficiency.