[Paper Review] Prioritized Experience Replay

Tom Schaul, John Quan+2 more|arXiv (Cornell University)|Nov 18, 2015

Neural dynamics and brain function29 references2,034 citations

TL;DR

The paper introduces prioritized experience replay, which samples more informative transitions (high TD-error) to improve learning efficiency, and demonstrates faster learning and state‑of‑the‑art Atari results when combined with Double DQN.

ABSTRACT

Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply replays transitions at the same frequency that they were originally experienced, regardless of their significance. In this paper we develop a framework for prioritizing experience, so as to replay important transitions more frequently, and therefore learn more efficiently. We use prioritized experience replay in Deep Q-Networks (DQN), a reinforcement learning algorithm that achieved human-level performance across many Atari games. DQN with prioritized experience replay achieves a new state-of-the-art, outperforming DQN with uniform replay on 41 out of 49 games.

Motivation & Objective

• Motivate experience replay inefficiencies in online RL and the potential gains from prioritization.
• Develop a stochastic prioritized replay mechanism with TD-error as the priority measure.
• Address biases and diversity issues with stochastic prioritization and importance sampling.
• Demonstrate scalability and effectiveness on the Atari 2600 benchmark using Double DQN.
• Outline extensions to supervised learning and off-policy scenarios.

Proposed method

• Define transition priority p_i as the TD-error magnitude |δ_i| or rank-based equivalents.
• Sample from replay memory using P(i) ∝ p_i^α to favor informative experiences (α controls prioritization).
• Use stochastic prioritization to preserve diversity and mitigate bias.
• Correct sampling bias with importance-sampling weights w_i and anneal β toward 1 over training.
• Integrate prioritized replay into Double DQN, replacing uniform sampling in the replay step.
• Provide efficient implementations (sum-tree or segment-based sampling) for large memories.

Experimental results

Research questions

• RQ1 Does prioritizing replay based on TD-error improve learning efficiency over uniform replay?
• RQ2 Can stochastic prioritization and IS correction maintain convergence and reduce bias in function approximation regimes?
• RQ3 How does prioritized replay interact with advanced RL architectures (e.g., Double DQN) on challenging benchmarks like Atari?
• RQ4 What are practical considerations (hyperparameters α, β, memory size) for large-scale applications?

Key findings

• Prioritized replay substantially speeds up learning and improves performance on Atari games.
• Adding prioritized replay to DQN yields improvements on 41 of 49 games and raises median performance.
• Combining prioritized replay with Double DQN achieves new state-of-the-art Atari performance, boosting both median and mean scores.
• Rank-based and proportional prioritization generally perform similarly and outperform uniform replay.
• Both variants reduce time to reach strong performance and help in games with slow initial learning.
• Annealing IS correction (β) balances bias and learning stability during training.