[Paper Review] Learning Fine Pinch-Grasp Skills using Tactile Sensing from A Few Real-world Demonstrations

Xiaofeng Mao, Yucheng Xu+6 more|arXiv (Cornell University)|Jul 10, 2023

Interactive and Immersive Displays4 citations

TL;DR

This paper proposes a data-efficient Learning from Demonstration (LfD) framework that leverages high-dimensional tactile sensing and proprioceptive feedback to learn fine dexterous pinch-grasp skills from a single real-world human demonstration. By using a convolutional autoencoder to extract tactile features and fusing them with proprioceptive states in a behavior cloning network, the method enables robust, contact-aware grasping and re-grasping on unseen objects, even under large external perturbations, demonstrating one-shot skill transfer with tactile-rich feedback.

ABSTRACT

Imitation learning for robot dexterous manipulation, especially with a real robot setup, typically requires a large number of demonstrations. In this paper, we present a data-efficient learning from demonstration framework which exploits the use of rich tactile sensing data and achieves fine bimanual pinch grasping. Specifically, we employ a convolutional autoencoder network that can effectively extract and encode high-dimensional tactile information. Further, We develop a framework that achieves efficient multi-sensor fusion for imitation learning, allowing the robot to learn contact-aware sensorimotor skills from demonstrations. Our comparision study against the framework without using encoded tactile features highlighted the effectiveness of incorporating rich contact information, which enabled dexterous bimanual grasping with active contact searching. Extensive experiments demonstrated the robustness of the fine pinch grasp policy directly learned from few-shot demonstration, including grasping of the same object with different initial poses, generalizing to ten unseen new objects, robust and firm grasping against external pushes, as well as contact-aware and reactive re-grasping in case of dropping objects under very large perturbations. Furthermore, the saliency map analysis method is used to describe weight distribution across various modalities during pinch grasping, confirming the effectiveness of our framework at leveraging multimodal information.

Motivation & Objective

• To develop a data-efficient LfD framework that enables dexterous bimanual manipulation using rich tactile feedback from real robot demonstrations.
• To address the challenge of learning fine motor skills from a single demonstration by integrating high-dimensional tactile data with proprioceptive state information.
• To overcome the limitations of visual-only or low-dimensional sensor feedback in one-shot learning by exploiting the discriminative power of soft tactile sensors.
• To enable robust grasping and autonomous re-grasping under external disturbances, emerging naturally from learned sensorimotor policies.
• To analyze modality contributions using saliency maps, revealing how tactile and proprioceptive feedback are weighted across different phases of grasp execution.

Proposed method

• A convolutional autoencoder is used to extract and compress high-dimensional tactile images from soft TacTip sensors into a low-dimensional latent space representation.
• The learned tactile features are fused with proprioceptive state features (joint angles, end-effector positions) to form a multimodal state representation.
• A behavior cloning network is trained to map the fused state-action pairs from a single human demonstration to a policy that replicates the demonstrated motor skills.
• The framework uses tactile image difference (change in contact pattern) as an additional input modality to enhance sensitivity to dynamic contact changes.
• Saliency map analysis is applied to compute the relative contribution weights of tactile and proprioceptive modalities across different grasp stages.
• The policy is trained end-to-end on real robot data, avoiding simulation and enabling direct transfer to hardware without domain randomization or large-scale data.

Figure 1: Autonomous dexterous grasping with soft tactile sensors, including pre-grasp, press, roll-lift, and firm grasp.

Experimental results

Research questions

• RQ1 Can a robot learn fine dexterous pinch-grasp skills from a single real-world demonstration using only tactile and proprioceptive feedback?
• RQ2 How do tactile and proprioceptive modalities contribute differently across distinct phases of the grasp (pre-grasp, press, roll-lift, stabilization)?
• RQ3 To what extent does tactile sensing improve one-shot learning performance compared to proprioception alone?
• RQ4 Can the learned policy generalize to unseen objects with varying diameters, masses, and materials?
• RQ5 Can the policy autonomously detect and recover from large external disturbances, such as object drops or pushes?

Key findings

• The framework successfully learned a stable and robust fine pinch-grasp policy from a single real-world demonstration using only tactile and proprioceptive feedback.
• The policy generalized to ten unseen objects with different diameters, masses, and materials, demonstrating strong zero-shot generalization within the same object category.
• The policy maintained firm and stable grasps even when subjected to large external pushes, showing resilience to dynamic disturbances.
• The system autonomously executed successful re-grasping after object drops, a behavior not explicitly demonstrated but emergent from the learned sensorimotor policy.
• Saliency map analysis revealed that tactile feedback became dominant during contact phases (pressing and lifting), while proprioception played a key role in coordination during pre-grasp and stabilization.
• The comparison study confirmed that tactile sensing is essential for one-shot learning of dexterous skills, as baseline methods without tactile input failed to learn the task effectively.

Figure 2: Architecture detailing the teleoperation system for demonstrations and the LfD framework.