[Paper Review] Touch100k: A Large-Scale Touch-Language-Vision Dataset for Touch-Centric Multimodal Representation

Ning Cheng, Changhao Guan+8 more|arXiv (Cornell University)|Jun 6, 2024

Social Robot Interaction and HRIPsychology3 citations

TL;DR

This paper introduces Touch100k, a 100k-sample touch-language-vision dataset with multi-granularity tactile descriptions, and proposes TLV-Link, a curriculum learning-based pretraining method that aligns tactile, visual, and language modalities for GelSight sensors. The method achieves state-of-the-art performance in tactile representation learning, with a 21.2% accuracy drop in zero-shot grasping when curriculum learning is removed, demonstrating its effectiveness in zero-shot touch understanding and material property identification.

ABSTRACT

Touch holds a pivotal position in enhancing the perceptual and interactive capabilities of both humans and robots. Despite its significance, current tactile research mainly focuses on visual and tactile modalities, overlooking the language domain. Inspired by this, we construct Touch100k, a paired touch-language-vision dataset at the scale of 100k, featuring tactile sensation descriptions in multiple granularities (i.e., sentence-level natural expressions with rich semantics, including contextual and dynamic relationships, and phrase-level descriptions capturing the key features of tactile sensations). Based on the dataset, we propose a pre-training method, Touch-Language-Vision Representation Learning through Curriculum Linking (TLV-Link, for short), inspired by the concept of curriculum learning. TLV-Link aims to learn a tactile representation for the GelSight sensor and capture the relationship between tactile, language, and visual modalities. We evaluate our representation's performance across two task categories (namely, material property identification and robot grasping prediction), focusing on tactile representation and zero-shot touch understanding. The experimental evaluation showcases the effectiveness of our representation. By enabling TLV-Link to achieve substantial improvements and establish a new state-of-the-art in touch-centric multimodal representation learning, Touch100k demonstrates its value as a valuable resource for research. Project page: https://cocacola-lab.github.io/Touch100k/.

Motivation & Objective

• To address the lack of large-scale, multi-granular language-annotated tactile datasets that integrate touch, language, and vision for robotics.
• To develop a pretraining method that learns robust tactile representations by aligning touch, language, and visual modalities.
• To enable zero-shot generalization in tactile understanding tasks such as material property identification and robot grasping prediction.
• To evaluate the impact of dataset scale and curriculum learning on tactile representation performance.

Proposed method

• Constructed Touch100k by collecting 101,982 visual-tactile observations from public datasets and using GPT-4V with prompt engineering to generate multi-granularity textual descriptions (sentence-level and phrase-level).
• Implemented a teacher-student curriculum learning framework where the vision encoder acts as a teacher to guide the tactile encoder (student) through a weighted curriculum representation combining visual and tactile features.
• Gradually reduced the teacher model's influence during pretraining as the student model's capacity improved, enabling progressive knowledge distillation.
• Fused multi-granularity language descriptions using a text encoder and applied contrastive learning to align the curriculum representation with the language modality.
• Trained the tactile encoder using joint learning on touch and vision in the first stage, followed by contrastive alignment with a frozen OpenCLIP-large text encoder in the second stage.
• Evaluated performance via linear probing and zero-shot transfer on material property identification and robot grasping prediction tasks.

Experimental results

Research questions

• RQ1 Can a large-scale, multi-granular touch-language-vision dataset improve zero-shot tactile understanding and representation learning?
• RQ2 How does curriculum learning enhance the generalization and performance of tactile representations in multimodal pretraining?
• RQ3 To what extent does dataset scale influence downstream performance in tactile representation learning?
• RQ4 How effective is the proposed TLV-Link method in capturing cross-modal relationships between touch, language, and vision?

Key findings

• TLV-Link achieved state-of-the-art performance in both linear probing and zero-shot evaluation across material property identification and robot grasping prediction tasks.
• Removing the curriculum representation led to a 21.2% drop in zero-shot grasping accuracy (from 65.4% to 44.2%), confirming its critical role in generalization.
• Reducing dataset size to 25% of the original (25k samples) had minimal impact on linear probing performance, indicating strong data efficiency.
• t-SNE visualization showed that TLV-Link effectively separates hard/soft and rough/smooth material categories, but struggles with multi-class and highly diverse distributions.
• The method outperformed existing baselines, with performance approaching the upper bound set by the TAG dataset features, validating its effectiveness.
• The ablation study confirmed that curriculum representation significantly improves performance, especially in zero-shot settings, highlighting its value in learning robust tactile representations.