[Paper Review] Learning Dexterous In-Hand Manipulation

OpenAI, Andrychowicz, Marcin+15 more|arXiv (Cornell University)|Aug 1, 2018

Robot Manipulation and LearningEngineering65 references277 citations

TL;DR

The paper trains a vision-based, memory-augmented reinforcement learning policy in a randomized simulator to perform dexterous in-hand object reorientation with the Shadow Hand, and transfers it to a real robot without human demonstrations.

ABSTRACT

We use reinforcement learning (RL) to learn dexterous in-hand manipulation policies which can perform vision-based object reorientation on a physical Shadow Dexterous Hand. The training is performed in a simulated environment in which we randomize many of the physical properties of the system like friction coefficients and an object's appearance. Our policies transfer to the physical robot despite being trained entirely in simulation. Our method does not rely on any human demonstrations, but many behaviors found in human manipulation emerge naturally, including finger gaiting, multi-finger coordination, and the controlled use of gravity. Our results were obtained using the same distributed RL system that was used to train OpenAI Five. We also include a video of our results: https://youtu.be/jwSbzNHGflM

Motivation & Objective

• Demonstrate vision-based, dexterous in-hand manipulation on a physical robot using entirely simulated training.
• Show that extensive randomization and memory-augmented policies enable transfer from simulation to reality.
• Develop a vision-based object pose estimator trained solely in simulation for deployment without markers.
• Evaluate the importance of randomizations and memory in achieving robust transfer.
• Analyze the sample complexity and scalability of the distributed RL system in training dexterous policies.

Proposed method

• Train a control policy as a memory-augmented recurrent network (LSTM) using PPO on a distribution of randomized MuJoCo simulations of the Shadow Hand manipulating objects.
• Discretize actions into 11 bins per joint to simplify learning.
• Use an asymmetric actor-critic setup where the value network has access to additional (simulated) information not available on the real robot.
• Randomize physical parameters (friction, masses, gravity, etc.) and visual appearances across episodes to bridge the reality gap (domain randomization).
• Train a separate vision model on synthetic renderings to predict object pose from 3 RGB camera views, enabling vision-based state estimation on the real robot.

Experimental results

Research questions

• RQ1 Can a policy trained entirely in simulation with heavy domain randomization transfer to a real five-finger dexterous hand performing in-hand object reorientation?
• RQ2 What role do memory, observation randomization, and physics randomization play in successful transfer?
• RQ3 Is a vision-based pose estimator trained only on synthetic data sufficient for real-world operation?
• RQ4 How does system scale with distributed RL training, and what is the impact of memory on performance?
• RQ5 What grasps and manipulation strategies emerge from learning without human demonstrations?

Key findings

• Policies exhibit human-like grasps and dexterous behaviors such as finger gaiting, multi-finger coordination, and gravity-aware manipulation without demonstrations.
• Training with extensive randomizations enables transfer to the real Shadow Hand, though real-world performance is lower than in simulation due to the reality gap.
• Vision-based pose estimation trained in simulation can be used to drive the policy on real hardware with performance close to marker-based pose estimates.
• A memory-enabled policy (LSTM) significantly improves transfer and performance over non-memory policies.
• The system scales with distributed RL, achieving substantial speedups with more GPUs/CPUs, and shows memory-augmented policies outperform non-memory variants on the physical robot.