[Paper Review] Spatio-temporal Attention Model for Tactile Texture Recognition

Guanqun Cao, Yi Zhou+2 more|arXiv (Cornell University)|Aug 10, 2020

Tactile and Sensory Interactions33 references4 citations

TL;DR

This paper proposes a novel Spatio-Temporal Attention Model (STAM) for tactile texture recognition that dynamically focuses on salient spatial regions and temporally relevant contact events in tactile sequences. By integrating spatial and temporal attention mechanisms, STAM improves recognition accuracy by up to 18.8% over non-attention baselines and demonstrates strong robustness to noisy pre-contact data, achieving a 15.23% average accuracy gain over CNN-based models when noise is introduced.

ABSTRACT

Recently, tactile sensing has attracted great interest in robotics, especially for facilitating exploration of unstructured environments and effective manipulation. A detailed understanding of the surface textures via tactile sensing is essential for many of these tasks. Previous works on texture recognition using camera based tactile sensors have been limited to treating all regions in one tactile image or all samples in one tactile sequence equally, which includes much irrelevant or redundant information. In this paper, we propose a novel Spatio-Temporal Attention Model (STAM) for tactile texture recognition, which is the very first of its kind to our best knowledge. The proposed STAM pays attention to both spatial focus of each single tactile texture and the temporal correlation of a tactile sequence. In the experiments to discriminate 100 different fabric textures, the spatially and temporally selective attention has resulted in a significant improvement of the recognition accuracy, by up to 18.8%, compared to the non-attention based models. Specifically, after introducing noisy data that is collected before the contact happens, our proposed STAM can learn the salient features efficiently and the accuracy can increase by 15.23% on average compared with the CNN based baseline approach. The improved tactile texture perception can be applied to facilitate robot tasks like grasping and manipulation.

Motivation & Objective

• To address the limitation of existing tactile texture recognition methods that treat all spatial regions and temporal samples equally, leading to inefficiency and noise sensitivity.
• To investigate the application of attention mechanisms—inspired by human tactile selective attention—in robotic tactile perception for the first time.
• To improve recognition accuracy and robustness by learning spatially and temporally selective features from tactile sequences.
• To enable more efficient learning from limited or noisy tactile data, particularly in real-world robotic manipulation scenarios.

Proposed method

• The STAM model employs a dual-branch architecture: a spatial attention module that assigns learnable weights to different regions of each tactile image to emphasize informative features.
• A temporal attention module is integrated to model long-range dependencies across sequential tactile samples, allowing the network to focus on the most relevant contact events over time.
• The spatial and temporal attention modules are jointly optimized using a cross-entropy loss function during end-to-end training on fabric texture data.
• Grad-CAM visualization is used to interpret and validate the spatial attention mechanism’s ability to highlight salient regions in tactile images.
• Temporal attention maps are visualized to analyze how past contact events influence the representation of current tactile states, particularly emphasizing recent inputs.
• The model is evaluated on a 100-class fabric texture dataset, with ablation studies on noisy data to assess robustness.

Experimental results

Research questions

• RQ1 Can a spatio-temporal attention mechanism improve tactile texture recognition accuracy by selectively focusing on informative spatial and temporal features?
• RQ2 How does the proposed STAM model perform under noisy conditions, particularly when pre-contact data is included?
• RQ3 To what extent can the attention mechanism enhance feature learning efficiency with limited input sequences?
• RQ4 Does the model’s attention distribution align with human-like tactile selective attention, favoring salient features and recent contact events?

Key findings

• The STAM model achieves a maximum accuracy improvement of 18.8% over non-attention baselines when recognizing 100 fabric textures, demonstrating the effectiveness of spatio-temporal attention.
• When noisy pre-contact data is introduced, the STAM model maintains high performance, with only a 0.92% accuracy drop (vs. 14.96% for the baseline), proving strong robustness.
• The STAM model improves recognition accuracy by 15.23% on average over a CNN-based baseline when trained on noisy data, highlighting its efficiency in filtering irrelevant information.
• Spatial attention visualization via Grad-CAM confirms that the model activates more informative tactile regions compared to the non-attention baseline, indicating better feature selection.
• Temporal attention analysis shows that the model consistently attends to the most recent contact regions, enabling effective modeling of long-range temporal dependencies.
• Despite having 11.4 million more parameters than the baseline (68.8M vs. 57.4M), the STAM remains compact and effective, indicating that the attention mechanism adds value without excessive complexity.