[Paper Review] Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations

Aravind Rajeswaran, Vikas Kumar+5 more|arXiv (Cornell University)|Sep 28, 2017

Robot Manipulation and Learning4 citations

TL;DR

This paper proposes DAPG, a deep reinforcement learning method that integrates human demonstrations into policy gradient training to enable sample-efficient learning of complex dexterous manipulation tasks with a 24-DoF robotic hand. By pre-training with behavior cloning and fine-tuning with a regularized loss, DAPG reduces sample complexity to under 5 hours of robot experience, yielding policies that are more robust and exhibit human-like motion compared to reward-shaping baselines.

ABSTRACT

Dexterous multi-fingered hands are extremely versatile and provide a generic way to perform a multitude of tasks in human-centric environments. However, effectively controlling them remains challenging due to their high dimensionality and large number of potential contacts. Deep reinforcement learning (DRL) provides a model-agnostic approach to control complex dynamical systems, but has not been shown to scale to high-dimensional dexterous manipulation. Furthermore, deployment of DRL on physical systems remains challenging due to sample inefficiency. Consequently, the success of DRL in robotics has thus far been limited to simpler manipulators and tasks. In this work, we show that model-free DRL can effectively scale up to complex manipulation tasks with a high-dimensional 24-DoF hand, and solve them from scratch in simulated experiments. Furthermore, with the use of a small number of human demonstrations, the sample complexity can be significantly reduced, which enables learning with sample sizes equivalent to a few hours of robot experience. The use of demonstrations result in policies that exhibit very natural movements and, surprisingly, are also substantially more robust.

Motivation & Objective

• To enable model-free deep reinforcement learning to scale to high-dimensional dexterous manipulation tasks with a 24-DoF robotic hand.
• To reduce the sample complexity of deep reinforcement learning for dexterous manipulation to levels feasible for real-world deployment.
• To improve policy robustness and naturalness by incorporating human priors through demonstrations.
• To establish a benchmark suite of complex dexterous manipulation tasks for future research in robotic manipulation and RL.

Proposed method

• Pre-train a policy using behavior cloning on a small set of human demonstrations collected via virtual reality.
• Fine-tune the policy using a variant of DDPG called DAPG, which incorporates demonstrations into the replay buffer and applies regularization to maintain proximity to demonstrated behavior.
• Use prioritized experience replay and n-step returns to improve sample efficiency and training stability.
• Augment the policy gradient objective with a regularization term that encourages the learned policy to stay close to the demonstrated trajectories.
• Train policies on an ensemble of environments with varying object properties (size and mass) to explicitly improve robustness.
• Evaluate performance on four complex tasks: object relocation, in-hand manipulation, tool use, and door opening, using sparse task completion rewards.

Experimental results

Research questions

• RQ1 Can model-free deep reinforcement learning solve complex dexterous manipulation tasks with a 24-DoF robotic hand without relying on hand-designed reward shaping?
• RQ2 How effective are human demonstrations in reducing sample complexity for learning dexterous manipulation policies in high-dimensional control settings?
• RQ3 Do policies trained with demonstrations exhibit more robustness and human-like motion compared to those trained from scratch with shaped rewards?
• RQ4 Can training on an ensemble of environments with varying physical properties further enhance policy robustness in dexterous manipulation?
• RQ5 Is the DAPG algorithm significantly more sample-efficient than existing DRL baselines like DDPGfD for high-dimensional dexterous manipulation tasks?

Key findings

• DAPG reduces sample complexity by up to 30 times compared to training from scratch with shaped rewards, enabling training in under 5 hours of robot experience.
• Policies trained with DAPG exhibit significantly higher robustness to environmental variations, such as changes in object mass and size, compared to those trained with shaped rewards.
• The use of demonstrations results in policies that produce smoother, more human-like motions, which are difficult to specify through reward shaping alone.
• DAPG outperforms DDPGfD across all four benchmark tasks, showing faster convergence and superior final performance.
• Training on an ensemble of environments further enhances robustness, and DAPG successfully learns policies for this challenging setting where baseline methods fail.
• The proposed benchmark suite of object relocation, in-hand manipulation, tool use, and door opening provides a realistic and challenging testbed for future dexterous manipulation research.