[Paper Review] Visuo-Tactile Transformers for Manipulation

Yizhou Chen, Andrea Sipos+2 more|arXiv (Cornell University)|Sep 30, 2022

Tactile and Sensory Interactions4 citations

TL;DR

This paper proposes Visuo-Tactile Transformers (VTT), a multimodal representation learning framework that extends Vision Transformers to jointly process visual and tactile feedback using cross-modal attention. VTT learns spatially-aware latent heatmaps that improve sample efficiency and performance in simulated and real-world robotic manipulation tasks, outperforming baselines in both simulated and real-world block pushing tasks.

ABSTRACT

Learning representations in the joint domain of vision and touch can improve manipulation dexterity, robustness, and sample-complexity by exploiting mutual information and complementary cues. Here, we present Visuo-Tactile Transformers (VTTs), a novel multimodal representation learning approach suited for model-based reinforcement learning and planning. Our approach extends the Visual Transformer \cite{dosovitskiy2021image} to handle visuo-tactile feedback. Specifically, VTT uses tactile feedback together with self and cross-modal attention to build latent heatmap representations that focus attention on important task features in the visual domain. We demonstrate the efficacy of VTT for representation learning with a comparative evaluation against baselines on four simulated robot tasks and one real world block pushing task. We conduct an ablation study over the components of VTT to highlight the importance of cross-modality in representation learning.

Motivation & Objective

• To improve robotic manipulation by learning compact, informative latent representations from joint visual and tactile feedback.
• To address the limitations of vector-based latent representations in capturing spatial details and local task-relevant features.
• To enhance sample efficiency and policy performance in model-based reinforcement learning by leveraging cross-modal attention.
• To enable more robust and stable policy learning through spatially-aware attention on task-relevant visual features guided by tactile signals.

Proposed method

• Extends the Vision Transformer (ViT) architecture to incorporate tactile feedback via modality-specific patching and embedding.
• Uses self-attention within visual and tactile modalities and cross-modal attention between them to build spatially distributed latent representations.
• Incorporates learned contact, alignment, and positional embeddings to improve multimodal reasoning and feature alignment.
• Applies a compression head to reduce the latent vector dimensionality, optimizing for efficiency and performance.
• Employs prediction and reconstruction losses on visual and tactile inputs, along with task-specific rewards, to supervise representation learning.
• Uses a multi-head attention mechanism with spatially grounded attention maps to highlight relevant visual regions based on tactile feedback.

Experimental results

Research questions

• RQ1 Can cross-modal attention between vision and touch improve representation quality for robotic manipulation?
• RQ2 How does spatially-aware attention in a multimodal latent space affect sample efficiency and task performance in RL?
• RQ3 Does integrating tactile feedback into a Vision Transformer architecture lead to better generalization and robustness compared to vector-based fusion methods?
• RQ4 How do contact and alignment losses contribute to the learning of effective visuo-tactile representations?
• RQ5 To what extent can VTT generalize to real-world robotic manipulation tasks with contact-rich interactions?

Key findings

• VTT outperformed both concatenation and Product of Experts (PoE) fusion baselines in simulated tasks, achieving higher success rates and faster learning convergence.
• In the real-world block pushing task, VTT achieved a higher success rate and demonstrated superior sample efficiency compared to baseline methods.
• Abllation studies showed that removing contact or alignment losses degraded performance, confirming their importance in learning effective multimodal representations.
• The optimal latent vector compression factor was found to be c=12, balancing representational capacity and learning efficiency.
• Attention heatmaps revealed that VTT dynamically focuses on the robot, object, and goal—especially during contact phases—demonstrating effective spatial localization guided by tactile feedback.
• The model successfully localized task-relevant features even when the object was far from the robot in the visual field, indicating robust spatial reasoning.