[Paper Review] Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards

Matej Vecerík, Todd Hester+8 more|arXiv (Cornell University)|Jul 27, 2017

Reinforcement Learning in Robotics20 references509 citations

TL;DR

The paper extends DDPG with demonstrations (DDPGfD) to learn from sparse rewards in robot insertion tasks, using a replay buffer with prioritized sampling, n-step returns, and multi-update training to outperform standard DDPG in both simulation and real hardware.

ABSTRACT

We propose a general and model-free approach for Reinforcement Learning (RL) on real robotics with sparse rewards. We build upon the Deep Deterministic Policy Gradient (DDPG) algorithm to use demonstrations. Both demonstrations and actual interactions are used to fill a replay buffer and the sampling ratio between demonstrations and transitions is automatically tuned via a prioritized replay mechanism. Typically, carefully engineered shaping rewards are required to enable the agents to efficiently explore on high dimensional control problems such as robotics. They are also required for model-based acceleration methods relying on local solvers such as iLQG (e.g. Guided Policy Search and Normalized Advantage Function). The demonstrations replace the need for carefully engineered rewards, and reduce the exploration problem encountered by classical RL approaches in these domains. Demonstrations are collected by a robot kinesthetically force-controlled by a human demonstrator. Results on four simulated insertion tasks show that DDPG from demonstrations out-performs DDPG, and does not require engineered rewards. Finally, we demonstrate the method on a real robotics task consisting of inserting a clip (flexible object) into a rigid object.

Motivation & Objective

• Demonstrate that demonstrations can replace reward shaping for challenging robotic manipulation tasks with sparse rewards.
• Integrate demonstrations into an off-policy RL framework to improve data efficiency and learning stability.
• Show that prioritized replay, n-step returns, and repeated updates enhance learning when demonstrations are used.
• Validate the approach on four simulated insertion tasks and one real-robot insertion task.

Proposed method

• Extend DDPG to incorporate demonstration transitions into the replay buffer before training begins.
• Use prioritized experience replay to sample both demonstration and agent transitions with a bias towards more informative experiences.
• Combine 1-step and n-step return losses for the critic to propagate sparse rewards over trajectories.
• Perform multiple gradient updates per environment step to improve data efficiency while maintaining stability.
• Apply L2 regularization on actor and critic networks for stability.
• Implement safety constraints in real-robot trials via an impedance controller to limit excessive forces.

Experimental results

Research questions

• RQ1 Can demonstrations replace hand-engineered shaping rewards in sparse-reward robotic insertion tasks?
• RQ2 Does integrating demonstrations into an off-policy framework with prioritized replay accelerate learning and improve performance over standard DDPG?
• RQ3 How do 1-step and n-step returns contribute to propagating sparse rewards in demonstration-augmented RL?
• RQ4 What is the effect of varying the amount of demonstration data on learning efficiency and final performance?
• RQ5 Are results consistent between simulated tasks and real-robot experiments?

Key findings

• DDPGfD outperforms DDPG across all tested tasks, even when DDPG uses well-tuned shaping rewards.
• DDPGfD learns effectively with sparse rewards, often matching or surpassing shaped-reward performance.
• In the clip insertion task, DDPGfD learns to insert 2–4x faster than demonstrations alone and with broader training stability.
• A single demonstration can enable solving the sparse-reward clip-insertion task, with diminishing returns after 50–100 demonstrations.
• Real-robot experiments show DDPGfD achieves robust insertion policies without engineered rewards, outperforming DDPG with shaping rewards.