[Paper Review] Advances in Experience Replay

Tracy Wan, Neil Xu|arXiv (Cornell University)|May 15, 2018

Online Learning and Analytics3 citations

TL;DR

This paper investigates the combination of three advanced experience replay techniques—Combined Experience Replay (CER), Prioritized Experience Replay (PER), and Hindsight Experience Replay (HER)—with DQN and DDPG algorithms across multiple OpenAI Gym environments. Results show that while HER significantly improves learning in sparse-reward environments like Pendulum-v0, combining all three techniques does not consistently outperform baselines, with performance highly dependent on environment structure and training stability.

ABSTRACT

This project combines recent advances in experience replay techniques, namely, Combined Experience Replay (CER), Prioritized Experience Replay (PER), and Hindsight Experience Replay (HER). We show the results of combinations of these techniques with DDPG and DQN methods. CER always adds the most recent experience to the batch. PER chooses which experiences should be replayed based on how beneficial they will be towards learning. HER learns from failure by substituting the desired goal with the achieved goal and recomputing the reward function. The effectiveness of combinations of these experience replay techniques is tested in a variety of OpenAI gym environments.

Motivation & Objective

• To evaluate the effectiveness of combining recent experience replay advances—CER, PER, and HER—for improving sample efficiency and learning stability in deep reinforcement learning.
• To assess whether synergistic integration of these techniques leads to superior convergence and performance over individual methods or standard experience replay.
• To test the robustness and generalization of combined experience replay across diverse reinforcement learning environments with varying reward sparsity and action space types.
• To identify which combinations yield measurable improvements in training speed and final performance on benchmark environments such as LunarLander-v2, MountainCar, and Pendulum-v0.
• To investigate the impact of hyperparameter sensitivity and training instability on the reliability of combined experience replay methods.

Proposed method

• Combines CER, which ensures the most recent experience is always included in training batches, to maintain temporal relevance.
• Integrates PER, which prioritizes replay of transitions with high temporal difference (TD) errors to focus learning on informative experiences.
• Applies HER, which redefines failed trajectories by replacing the original goal with the achieved goal and re-computing the reward, enabling learning from failure.
• Implements the combined approach within DQN and DDPG frameworks, using replay buffers that store transitions with state, action, reward, next state, and goal information.
• Uses target networks and soft updates in DDPG to stabilize training, while applying batch normalization to handle varying state and action scales.
• Employs exploration via noise injection in DDPG and evaluates performance across multiple random seeds to assess variance and stability.

Experimental results

Research questions

• RQ1 Does combining CER, PER, and HER lead to faster convergence and higher final performance compared to baseline experience replay in DQN and DDPG?
• RQ2 How do these combined techniques perform in environments with sparse rewards (e.g., MountainCar, Pendulum) versus dense-reward environments (e.g., LunarLander)?
• RQ3 To what extent does the inclusion of HER improve learning in continuous control tasks where failure is common?
• RQ4 Does the combination of prioritization (PER) and hindsight learning (HER) amplify learning efficiency, or do they interfere due to conflicting sampling objectives?
• RQ5 How stable and reproducible are the results of combined experience replay across multiple training runs and environments?

Key findings

• HER significantly improved learning speed and success rate in the Pendulum-v0 environment, with CHER (CER + HER) converging in just 500 episodes, outperforming other combinations.
• In LunarLander-v2, which features dense rewards, techniques involving HER and PER performed worse than the baseline, suggesting these methods may be less effective in dense-reward settings.
• Combined experience replay (CER) alone showed performance close to baseline in LunarLander but did not surpass it, indicating limited incremental benefit in dense-reward environments.
• MountainCar results showed minimal differentiation across methods, with no clear advantage for any combination, likely due to the extreme sparsity of rewards and model instability.
• DDPG training was highly unstable across environments, particularly in Pendulum-v0, limiting the reliability of quantitative comparisons despite promising trends in HER-based methods.
• The combination of HER with PER and CER did not yield consistent improvements across all environments, highlighting the challenge of generalizing experience replay enhancements across diverse RL tasks.