[Paper Review] Curiosity-Driven Experience Prioritization via Density Estimation

Rui Zhao, Volker Tresp|arXiv (Cornell University)|Feb 20, 2019

Visual Attention and Saliency Detection49 references42 citations

TL;DR

CDP estimates trajectory densities in the replay buffer via a Variational Gaussian Mixture Model and prioritizes low-density (rare) achieved-goal trajectories to improve sample efficiency and performance when combined with off-policy RL methods like DDPG and DDPG+HER.

ABSTRACT

In Reinforcement Learning (RL), an agent explores the environment and collects trajectories into the memory buffer for later learning. However, the collected trajectories can easily be imbalanced with respect to the achieved goal states. The problem of learning from imbalanced data is a well-known problem in supervised learning, but has not yet been thoroughly researched in RL. To address this problem, we propose a novel Curiosity-Driven Prioritization (CDP) framework to encourage the agent to over-sample those trajectories that have rare achieved goal states. The CDP framework mimics the human learning process and focuses more on relatively uncommon events. We evaluate our methods using the robotic environment provided by OpenAI Gym. The environment contains six robot manipulation tasks. In our experiments, we combined CDP with Deep Deterministic Policy Gradient (DDPG) with or without Hindsight Experience Replay (HER). The experimental results show that CDP improves both performance and sample-efficiency of reinforcement learning agents, compared to state-of-the-art methods.

Motivation & Objective

• Motivate and address memory imbalance in replay buffers in RL by prioritizing under-represented achieved-goal trajectories.
• Develop a curiosity-inspired prioritization framework (CDP) that balances sampling in the experience buffer.
• Enable CDP to be combined with off-policy RL algorithms such as DDPG and DDPG+HER.
• Leverage density estimation to guide prioritized replay rather than TD-errors.
• Demonstrate improvements in sample efficiency and final performance on multi-task robotic manipulation tasks.

Proposed method

• Represent each trajectory by the concatenated sequence of goal states across time.
• Estimate trajectory density with a Variational Gaussian Mixture Model (V-GMM) trained on memory buffer data.
• Compute trajectory density rho and its complement bar{rho} = 1 - rho to identify rare trajectories.
• Rank trajectories by the rank of bar{rho} and sample according to this ranking-based probability to replay less frequent trajectories.
• Integrate CDP with off-policy RL algorithms (e.g., DDPG, DDPG+HER) and update the density model every epoch.
• Optionally compare CDP with Prioritized Experience Replay (PER) and show CDP’s favorable time complexity.

Experimental results

Research questions

• RQ1 Does incorporating CDP improve performance and sample efficiency for DDPG and DDPG+HER on multi-goal robotic tasks?
• RQ2 How does CDP compare to PER in terms of performance gains and computational efficiency?
• RQ3 What is the relationship between the density-based prioritization (bar{rho}) and TD-errors during training?

Key findings

• CDP accelerates convergence across all six robotic tasks and yields higher final success rates than baselines and PER.
• CDP improves sample efficiency by approximately a factor of two on average across the six environments.
• CDP is significantly faster than PER in training time (CDP ~ between baseline and PER; PER much slower), while delivering equal or better performance.
• The average improvement in final performance over baselines is about 9.15 percentage points.
• There is a positive correlation (average Pearson's r ≈ 0.7) between the complementary density bar{rho} and TD-errors, indicating rarer trajectories tend to be more valuable for learning.