[Paper Review] Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation

Sridhar Pandian Arunachalam, Sneha Silwal+2 more|arXiv (Cornell University)|Mar 24, 2022

Robot Manipulation and Learning5 citations

TL;DR

DIME is a learning-based framework that enables efficient dexterous manipulation using low-cost, single RGB camera teleoperation with untrained human operators. By leveraging off-the-shelf hand pose estimation and retargeting, it collects high-quality demonstrations that enable sample-efficient imitation learning, achieving high success rates on complex in-hand tasks like flipping, spinning, and rotating objects with the Allegro hand in both simulation and real-world settings.

ABSTRACT

Optimizing behaviors for dexterous manipulation has been a longstanding challenge in robotics, with a variety of methods from model-based control to model-free reinforcement learning having been previously explored in literature. Perhaps one of the most powerful techniques to learn complex manipulation strategies is imitation learning. However, collecting and learning from demonstrations in dexterous manipulation is quite challenging. The complex, high-dimensional action-space involved with multi-finger control often leads to poor sample efficiency of learning-based methods. In this work, we propose 'Dexterous Imitation Made Easy' (DIME) a new imitation learning framework for dexterous manipulation. DIME only requires a single RGB camera to observe a human operator and teleoperate our robotic hand. Once demonstrations are collected, DIME employs standard imitation learning methods to train dexterous manipulation policies. On both simulation and real robot benchmarks we demonstrate that DIME can be used to solve complex, in-hand manipulation tasks such as 'flipping', 'spinning', and 'rotating' objects with the Allegro hand. Our framework along with pre-collected demonstrations is publicly available at https://nyu-robot-learning.github.io/dime.

Motivation & Objective

• To address the challenge of sample inefficiency in learning dexterous manipulation policies for high-dimensional robotic hands.
• To reduce the barrier to collecting high-quality demonstrations by enabling untrained human operators to teleoperate robotic hands using only a single RGB camera.
• To develop a framework compatible with diverse imitation learning paradigms, including model-free RL and non-parametric methods, for both simulation and real-world deployment.
• To demonstrate that inexpensive, easily collected demonstrations can enable high-performance dexterous manipulation without specialized hardware or extensive calibration.

Proposed method

• Utilizes a single RGB camera to capture human hand movements and applies off-the-shelf hand pose detectors to estimate fingertip positions in real time.
• Retargets the estimated human fingertip positions to the robotic hand’s kinematic configuration using a kinematically consistent mapping.
• Employs real-time visual feedback to the human operator to enable intuitive, low-latency teleoperation with minimal training.
• Collects demonstrations using this pipeline, which are then used to train policies via imitation learning techniques such as behavior cloning, BCRL, DAPG, and PPO in simulation.
• Applies non-parametric nearest-neighbor learning (INN) and visual imitation (VINN) on real robot hardware using state and image-based representations, respectively.
• Uses self-supervised representation learning (BYOL) to improve visual feature quality for image-based imitation in cluttered scenes.

Experimental results

Research questions

• RQ1 Can dexterous manipulation policies be trained efficiently using only a single RGB camera and untrained human operators?
• RQ2 Can demonstrations collected via a low-cost, minimal-calibration teleoperation system be effective for imitation learning across both simulation and real-world deployment?
• RQ3 How do different imitation learning methods—parametric (e.g., BC, PPO) and non-parametric (e.g., INN)—perform when trained on demonstrations from a simple teleoperation pipeline?
• RQ4 What is the impact of representation learning (e.g., BYOL) on visual imitation performance in complex, cluttered scenes with limited demonstrations?

Key findings

• DIME enables demonstration collection in approximately 100 seconds per task, requiring minimal human training and no specialized hardware.
• In simulation, BCRL and PPO finetuning on demonstrations achieved smooth, human-like policies that outperformed raw teleop demonstrations, with PPO solving spinning and rotation tasks despite not being explicitly designed for them.
• On the real Allegro hand, INN achieved 100% success rate on 90-degree rotation and 80% on flipping, outperforming parametric behavior cloning, which failed entirely across all tasks.
• VINN achieved 90% success on flipping and 70% on 90-degree rotation using image-based imitation, though performance dropped on spinning due to visual complexity and poor representation learning in cluttered scenes.
• Pure behavior cloning failed on all tasks due to distributional shift and limited demonstration coverage, highlighting the necessity of RL finetuning.
• The framework is publicly released, including hand controllers, demonstration data, and learning code, to support future research in sample-efficient dexterous manipulation.