[Paper Review] Data-efficient Deep Reinforcement Learning for Dexterous Manipulation

Ivaylo Popov, Nicolas Heess+8 more|arXiv (Cornell University)|Apr 10, 2017

Reinforcement Learning in Robotics25 references117 citations

TL;DR

The paper extends DDPG with data-efficient, scalable methods—distributed asynchronous learning, variable replay steps, and reward/shaping strategies—to enable end-to-end dexterous manipulation in simulation, achieving the full Lego stacking task with under 10 million environment transitions and potentially on real robots.

ABSTRACT

Deep learning and reinforcement learning methods have recently been used to solve a variety of problems in continuous control domains. An obvious application of these techniques is dexterous manipulation tasks in robotics which are difficult to solve using traditional control theory or hand-engineered approaches. One example of such a task is to grasp an object and precisely stack it on another. Solving this difficult and practically relevant problem in the real world is an important long-term goal for the field of robotics. Here we take a step towards this goal by examining the problem in simulation and providing models and techniques aimed at solving it. We introduce two extensions to the Deep Deterministic Policy Gradient algorithm (DDPG), a model-free Q-learning based method, which make it significantly more data-efficient and scalable. Our results show that by making extensive use of off-policy data and replay, it is possible to find control policies that robustly grasp objects and stack them. Further, our results hint that it may soon be feasible to train successful stacking policies by collecting interactions on real robots.

Motivation & Objective

• Demonstrate end-to-end learning of dexterous manipulation in simulation using a general-purpose model-free RL algorithm.
• Improve data efficiency and scalability through replay scheduling and distributed training.
• Provide practical guidance on reward shaping and curriculum-like initial state distributions to solve complex compositional tasks.

Proposed method

• Extend Deep Deterministic Policy Gradient (DDPG) with two data-efficiency improvements: independent scheduling of network updates (replay steps) and a distributed asynchronous implementation (ADPG-R) across multiple workers.
• Introduce asynchronous DPG variants (ADPG-R) to parallelize data collection and network updates with shared parameters across workers.
• Evaluate effect of varying the number of mini-batch replay updates per environment step on learning speed and data efficiency.
• Propose two general strategies to inject prior knowledge: (1) composite shaping rewards for compositional tasks, and (2) learning from instructive starting states to create a curriculum-like learning signal.
• Provide experimental setup details in MuJoCo with a 9-DoF arm learning to grasp and stack Lego bricks, including observation, action spaces, and evaluation protocol.

Experimental results

Research questions

• RQ1 Can data-efficient, off-policy deep RL be used to solve complex dexterous manipulation tasks end-to-end in simulation?
• RQ2 How do replay-update schedules and distributed training affect data and wall-clock efficiency in learning manipulation policies?
• RQ3 Do composite shaping rewards and instructed starting states improve learning viability for multi-subtask manipulation (reach, grasp, stack) compared to sparse rewards?
• RQ4 What are practical guidelines for scaling data collection and learning to multiple robots in dexterous tasks?

Key findings

• Learning the full stacking task from scratch in less than 10 million environment transitions (under 10 hours on 16 robots) is feasible in simulation.
• Using more than one replay update per environment step dramatically improves data efficiency, enabling stacking after 200k–300k interactions in some settings.
• Asynchronous ADPG-R with multiple workers can significantly reduce wall-clock time while maintaining data efficiency, with speedups varying by task.
• Composite shaping rewards are necessary for solving the full Stack task; sparse rewards alone fail, and adding intermediate rewards (grasp, reach) accelerates learning, with full composition required for success.
• Learning from instructive start states (e.g., starting closer to subgoals or along solution trajectories) further improves exploration and can enable policy learning from varied initial conditions.