[Paper Review] A Deeper Look at Experience Replay

Shangtong Zhang, Richard S. Sutton|arXiv (Cornell University)|Dec 4, 2017

Reinforcement Learning in Robotics12 references185 citations

TL;DR

The paper systematically analyzes how replay buffer size affects learning across tabular, linear, and non-linear function approximators, showing large buffers can hurt performance and introducing a simple O(1) CER remedy to mitigate this effect.

ABSTRACT

Recently experience replay is widely used in various deep reinforcement learning (RL) algorithms, in this paper we rethink the utility of experience replay. It introduces a new hyper-parameter, the memory buffer size, which needs carefully tuning. However unfortunately the importance of this new hyper-parameter has been underestimated in the community for a long time. In this paper we did a systematic empirical study of experience replay under various function representations. We showcase that a large replay buffer can significantly hurt the performance. Moreover, we propose a simple O(1) method to remedy the negative influence of a large replay buffer. We showcase its utility in both simple grid world and challenging domains like Atari games.

Motivation & Objective

• Assess how replay buffer size impacts learning under different function representations (tabular, linear, non-linear).
• Identify conditions under which large replay buffers degrade performance.
• Propose a low-cost method to mitigate negative effects of large replay buffers without extra computational burden.
• Evaluate CER across simple and challenging domains (grid world, Lunar Lander, Atari Pong).

Proposed method

• Compare Online-Q, Buffer-Q, and Combined-Q (CER) in three tasks.
• Use uniform sampling from replay buffers, with CER adding the latest transition to each training batch.
• Evaluate across varying replay buffer sizes (e.g., 10^2 to 10^6) and fixed mini-batch size.
• Apply tabular, tile-coded linear, and neural network function approximators with appropriate optimizers.
• In non-linear settings, employ a target network for stability similar to DQN.
• Use timeouts and partial-episode-bootstrap to reduce timeout bias.

Experimental results

Research questions

• RQ1 How does the size of the replay buffer influence learning speed and final performance across different function representations?
• RQ2 Can a simple, constant-time modification (CER) reduce the negative impact of large replay buffers?
• RQ3 Do CER benefits persist across grid world, Lunar Lander, and Pong tasks with varying function approximators?

Key findings

• Large replay buffers can slow learning or impair performance in tabular and linear settings.
• CER makes learning less sensitive to replay buffer size and can recover speed in large-buffer regimes.
• Non-linear function approximation shows CER speeds up learning with large buffers and reduces overfitting to recent transitions.
• In some tasks (Pong), CER provides limited improvement, indicating task difficulty and representation affect CER benefIts.
• Across domains, a medium buffer can outperform very large buffers without CER.
• CER requires only O(1) extra computation and acts as a simple plug-in improvement.