[Paper Review] Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger

Arthur Allshire, Mayank Mittal+8 more|arXiv (Cornell University)|Aug 22, 2021

Robot Manipulation and Learning25 references15 citations

TL;DR

This paper presents a unified deep reinforcement learning policy for dexterous 6-DoF in-hand manipulation on the TriFinger robot, trained in GPU-accelerated simulation with domain randomization and keypoint-based observations and rewards. The method achieves 83% success rate in sim-to-real transfer on a remote real-world TriFinger, generalizing to unseen objects and enabling robust grasping, ungrasping, and repositioning with a single policy.

ABSTRACT

We present a system for learning a challenging dexterous manipulation task involving moving a cube to an arbitrary 6-DoF pose with only 3-fingers trained with NVIDIA's IsaacGym simulator. We show empirical benefits, both in simulation and sim-to-real transfer, of using keypoints as opposed to position+quaternion representations for the object pose in 6-DoF for policy observations and in reward calculation to train a model-free reinforcement learning agent. By utilizing domain randomization strategies along with the keypoint representation of the pose of the manipulated object, we achieve a high success rate of 83% on a remote TriFinger system maintained by the organizers of the Real Robot Challenge. With the aim of assisting further research in learning in-hand manipulation, we make the codebase of our system, along with trained checkpoints that come with billions of steps of experience available, at https://s2r2-ig.github.io

Motivation & Objective

• To develop a robust, unified reinforcement learning policy for complex in-hand manipulation tasks involving arbitrary 6-DoF object repositioning on a real-world robotic hand.
• To address the challenge of sim-to-real transfer in dexterous manipulation by leveraging domain randomization and high-fidelity simulation.
• To improve policy generalization across unseen objects and physical parameter variations through scalable training and keypoint-based representations.
• To enable deployment on remote, cloud-based robot systems without physical access, demonstrating robustness across fleet-wide deployments.
• To provide a reproducible, open-source framework with trained checkpoints to accelerate future research in sim-to-real dexterous manipulation.

Proposed method

• Trained a model-free deep reinforcement learning agent in a high-speed GPU-accelerated MuJoCo simulation environment with domain randomization for robust sim-to-real transfer.
• Used keypoint-based representations for object pose in both policy observations and dense reward functions to improve learning efficiency and generalization in 6-DoF manipulation.
• Employed large-scale training over billions of environment steps to learn a single unified policy capable of grasping, ungrasping, and repositioning objects.
• Integrated a remote robot-in-the-loop training loop using a cloud-based TriFinger robot farm to validate real-world performance without physical access.
• Applied fine-tuning on new object shapes and sizes to improve zero-shot generalization, especially for varying scales.
• Used off-the-shelf pose estimation (e.g., YOLO-based detectors) in inference to enable real-world deployment with minimal retraining.

(a) OpenAI’s shadow hand setup. The cube starts placed in the hand.

Experimental results

Research questions

• RQ1 Can a single deep reinforcement learning policy successfully learn to perform full 6-DoF in-hand manipulation, including grasping and repositioning, in simulation and transfer robustly to the real world?
• RQ2 How does using keypoint-based object pose representations in observations and rewards compare to traditional position+quaternion representations in terms of sim-to-real success rate and training stability?
• RQ3 To what extent can a single policy generalize to unseen object morphologies and scales without fine-tuning?
• RQ4 How effective is domain randomization in enabling high sim-to-real transfer performance on a remote, cloud-based robotic platform?
• RQ5 Can the policy recover from failures such as object drops and re-grasp objects autonomously in real-world deployment?

Key findings

• The policy using keypoint-based observations and rewards (O-KP+R-KP) achieved a 82.5% success rate on the real TriFinger robot, outperforming all ablations.
• Policies trained with keypoint-based rewards showed significantly higher success rates (77.5% and 82.5%) compared to those using position+quaternion-based rewards (55% and 60%).
• Zero-shot generalization to unseen cuboids of different sizes achieved a 46.8% success rate, which improved to 72.9% after fine-tuning, indicating scale is a more critical factor than shape for generalization.
• The policy demonstrated emergent behaviors such as 'dropping and regrasping' to improve pose accuracy and stability, especially in challenging target positions.
• The system successfully transferred from simulation to a remote real-world TriFinger robot with no physical access, validating robustness in a fleet-wide, cloud-based deployment setting.
• Fine-tuning on unseen objects improved success rates from 84.9% (zero-shot) to 87.2% on the EGAD benchmark, confirming the effectiveness of fine-tuning for domain shift mitigation.

(b) The Trifinger setup. The object starts outside of the hand to enable reset-free setup.