[Paper Review] DexTac: Learning Contact-aware Visuotactile Policies via Hand-by-hand Teaching

Xingyu Zhang, Chaofan Zhang+4 more|arXiv (Cornell University)|Jan 29, 2026

Tactile and Sensory Interactions0 citations

TL;DR

DexTac combines hand-by-hand kinesthetic demonstrations with multi-dimensional visuotactile sensing to learn contact-aware policies that predict both force and contact region (CoP) for dexterous manipulation, achieving high success in syringe injection tasks. It demonstrates strong generalization and data efficiency, and a tactile controller enforces contact-aware execution.

ABSTRACT

For contact-intensive tasks, the ability to generate policies that produce comprehensive tactile-aware motions is essential. However, existing data collection and skill learning systems for dexterous manipulation often suffer from low-dimensional tactile information. To address this limitation, we propose DexTac, a visuo-tactile manipulation learning framework based on kinesthetic teaching. DexTac captures multi-dimensional tactile data-including contact force distributions and spatial contact regions-directly from human demonstrations. By integrating these rich tactile modalities into a policy network, the resulting contact-aware agent enables a dexterous hand to autonomously select and maintain optimal contact regions during complex interactions. We evaluate our framework on a challenging unimanual injection task. Experimental results demonstrate that DexTac achieves a 91.67% success rate. Notably, in high-precision scenarios involving small-scale syringes, our approach outperforms force-only baselines by 31.67%. These results underscore that learning multi-dimensional tactile priors from human demonstrations is critical for achieving robust, human-like dexterous manipulation in contact-rich environments.

Motivation & Objective

• Provide high-quality multimodal expert demonstrations capturing joint states, RGB visuals, and tactile data (including contact region information).
• Learn a contact-aware policy that predicts both forces and contact regions (CoP) during dexterous manipulation.
• Deploy a tactile controller that combines policy predictions with tactile cues to control finger forces and contact regions in real time.
• Demonstrate improved success in a challenging unimanual syringe injection task and show generalization to unseen syringe sizes.

Proposed method

• Collect multimodal expert data via hand-by-hand kinesthetic teaching with fingertip sheaths to occlude human fingers and reduce domain shift.
• Capture RGB images, tactile imagery from GelStereo BioTip sensors, joint states, and interaction forces plus Center of Pressure (CoP) per fingertip.
• Train an ACT-based policy network that maps multimodal states to action chunks including joint variations, tactile forces, and CoP.
• Incorporate a latent variable z with a KL-divergence term to regularize the policy latent space and enable sequence-long planning.
• Deploy with a tactile controller that adjusts a force-informed position reference using both predicted forces and CoP to ensure contact-aware motion.
• Formulate policy learning as imperfect imitation learning with a loss that minimizes L1 error on action chunks and includes a KL regularization term.

Experimental results

Research questions

• RQ1 Does incorporating tactile information, especially CoP, improve dexterous manipulation performance in the injection task?
• RQ2 How well do the contact-aware policies generalize to unseen syringe sizes (zero-shot transfer)?
• RQ3 How does the performance scale with increasing amounts of demonstration data (data efficiency)?

Key findings

• DexTac achieves 91.67% average success across syringe sizes, outperforming force-only baselines by 31.67%.
• Zero-shot transfer to a 20 mL syringe yields 65% success, showing strong generalization.
• Ablation shows removing tactile sensing or CoP degrades performance, with tactile+CoP yielding the highest success.
• Data efficiency shows performance improves with more demonstrations, plateauing around 30 demonstrations per size.
• Purely tactile experiments indicate vision is beneficial in dynamic grasp-and-push tasks, while continuous contact cases can perform with minimal visual input.