[Paper Review] Distributed Prioritized Experience Replay

Dan Horgan, John Quan+5 more|arXiv (Cornell University)|Mar 2, 2018

Reinforcement Learning in Robotics25 references411 citations

TL;DR

The paper introduces Ape-X, a distributed architecture that decouples acting from learning and uses centralized prioritized experience replay to scale deep RL, achieving state-of-the-art Atari results and strong continuous control performance.

ABSTRACT

We propose a distributed architecture for deep reinforcement learning at scale, that enables agents to learn effectively from orders of magnitude more data than previously possible. The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared experience replay memory; the learner replays samples of experience and updates the neural network. The architecture relies on prioritized experience replay to focus only on the most significant data generated by the actors. Our architecture substantially improves the state of the art on the Arcade Learning Environment, achieving better final performance in a fraction of the wall-clock training time.

Motivation & Objective

• Motivate scaling deep reinforcement learning by increasing data generation and selective experience replay.
• Propose a distributed architecture that decouples acting from learning and uses a centralized replay with prioritization.
• Demonstrate scalability and performance gains on Atari and continuous control benchmarks.
• Analyze factors affecting scalability, such as replay capacity, recency, and policy diversity.

Proposed method

• Introduce Ape-X architecture with multiple actors collecting experience in parallel and a single learner updating network parameters.
• Use a centralized replay memory with proportional prioritization to sample informative experiences.
• Compute priorities online at the actor side to avoid delays in priority updates.
• Apply off-policy learning using variants of DQN (with double Q-learning, multi-step returns, dueling networks) and DDPG (Ape-X DPG).
• Actors periodically fetch latest parameters from the learner; learning updates and priority updates run asynchronously.
• Evaluate on Atari with 360 actors and on continuous control tasks from the DeepMind Control Suite.
• Report scalability analyses varying number of actors, replay capacity, recency, and data-generating policies.

Experimental results

Research questions

• RQ1 Can a distributed architecture that decouples data generation from learning and uses prioritized replay improve sample efficiency and final performance in deep RL?
• RQ2 How does the number of data-generating actors, replay memory capacity, and policy diversity affect scaling and performance on discrete and continuous control tasks?
• RQ3 Does online priority computation at the actor level aid scalability without slowing data generation?
• RQ4 Can Ape-X achieve state-of-the-art results on Atari and competitive performance on continuous control benchmarks without per-game hyperparameter tuning.

Key findings

• Ape-X achieves state-of-the-art median human-normalized scores across 57 Atari games, with faster wall-clock training and higher final performance than baselines.
• Scaling the number of actors consistently improves performance on Atari: from 8 to 256 actors, while keeping learner updates fixed.
• Larger replay memory capacity yields marginal to moderate gains, suggesting benefits from retaining high-priority experiences over time.
• Prioritized replay, combined with diverse behavior policies across many actors, helps exploration and avoids overfitting, leading to better performance.
• Ape-X DQN with double Q-learning, multi-step returns, and dueling networks shows strong Atari performance; Ape-X DPG extends to continuous control with competitive results across tasks.
• In continuous control, increasing the number of actors improves rapid learning and final performance beyond standard DDPG baselines.