[Paper Review] H-InDex: Visual Reinforcement Learning with Hand-Informed Representations for Dexterous Manipulation

Yanjie Ze, Yuyao Liu+5 more|arXiv (Cornell University)|Oct 2, 2023

Robot Manipulation and Learning4 citations

TL;DR

H-InDex proposes a visual reinforcement learning framework that leverages pre-trained 3D human hand pose representations to enhance robotic dexterous manipulation. By fine-tuning only 0.36% of parameters via self-supervised keypoint detection and EMA BatchNorm adaptation, it achieves state-of-the-art performance across 12 challenging manipulation tasks, outperforming recent visual foundation models.

ABSTRACT

Human hands possess remarkable dexterity and have long served as a source of inspiration for robotic manipulation. In this work, we propose a human $ extbf{H}$and$ extbf{-In}$formed visual representation learning framework to solve difficult $ extbf{Dex}$terous manipulation tasks ($ extbf{H-InDex}$) with reinforcement learning. Our framework consists of three stages: (i) pre-training representations with 3D human hand pose estimation, (ii) offline adapting representations with self-supervised keypoint detection, and (iii) reinforcement learning with exponential moving average BatchNorm. The last two stages only modify $0.36\%$ parameters of the pre-trained representation in total, ensuring the knowledge from pre-training is maintained to the full extent. We empirically study 12 challenging dexterous manipulation tasks and find that H-InDex largely surpasses strong baseline methods and the recent visual foundation models for motor control. Code is available at https://yanjieze.com/H-InDex .

Motivation & Objective

• To improve dexterous manipulation in robotic hands using visual representations derived from human hand dexterity.
• To address the challenge of transferring human hand knowledge to robotic policy learning in visual reinforcement learning.
• To minimize catastrophic forgetting during adaptation by updating only a small fraction of parameters in pre-trained models.
• To demonstrate that off-the-shelf human hand pose estimators can serve as effective visual priors for robotic control without full retraining.

Proposed method

• Pre-trains visual representations using a 3D human hand pose estimator (FrankMocap) to embed human dexterity into the feature encoder.
• Adapts the pre-trained representation via self-supervised keypoint detection, updating only 0.18% of parameters (affine transformations in BatchNorm layers).
• Applies exponential moving average (EMA) BatchNorm during reinforcement learning to dynamically adjust normalization statistics for better adaptation.
• Maintains the convolutional backbone frozen during adaptation, preserving pre-trained knowledge while enabling domain shift generalization.
• Uses a standard visual RL pipeline with the adapted representation as input, keeping policy and reward design unchanged for orthogonal evaluation.
• Employs a three-stage pipeline: (1) pre-training on human hand pose, (2) offline adaptation with minimal fine-tuning, (3) end-to-end RL training with EMA BatchNorm.

Figure 1 : Normalized average score for our algorithm H-InDex and the baselines (VC-1 [ 17 ] , MVP [ 31 ] , R3M [ 18 ] , and RRL [ 27 ] ).

Experimental results

Research questions

• RQ1 Can pre-trained 3D human hand pose representations improve sample efficiency and performance in robotic dexterous manipulation tasks?
• RQ2 Does fine-tuning only a small subset of parameters (0.18%) in a pre-trained visual model preserve human dexterity while adapting to robotic domains?
• RQ3 How does EMA BatchNorm improve policy training stability and performance in visual RL with adapted representations?
• RQ4 Can a model pre-trained for human hand understanding generalize effectively to robotic manipulation tasks without task-specific pre-training?
• RQ5 How does H-InDex compare to recent visual foundation models like VC-1, MVP, and R3M in terms of performance and robustness?

Key findings

• H-InDex outperforms strong baselines including VC-1, MVP, R3M, and RRL across 12 challenging dexterous manipulation tasks, as shown in normalized average score comparisons.
• The method achieves superior performance by preserving 99.64% of the pre-trained model’s parameters, minimizing catastrophic forgetting during adaptation.
• Adapting only 0.18% of parameters (BatchNorm affine weights) yields better results than fine-tuning all parameters, confirming the benefit of parameter-efficient adaptation.
• EMA BatchNorm in Stage 3 significantly improves training stability and final performance, as ablation studies confirm its necessity.
• H-InDex generalizes better than VC-1 to novel backgrounds, as demonstrated in the relocate potted meat can task with changed scene backgrounds.
• Visualization confirms that the adapted model successfully attends to hand and object keypoints, even with minimal parameter updates, indicating effective feature alignment.

Figure 2 : The overview of H-InDex. H-InDex consists of three stages: 1) representation pre-training, 2) representation offline adaptation, and 3) reinforcement learning.