[Paper Review] Towards Learning to Detect and Predict Contact Events on Vision-based Tactile Sensors

Yazhan Zhang, Weihao Yuan+2 more|arXiv (Cornell University)|Oct 9, 2019

EEG and Brain-Computer Interfaces20 citations

TL;DR

This paper proposes a deep learning framework, PixelMotionNet, for detecting and predicting tactile contact events using vision-based tactile sensors. By collecting 6,650 labeled tactile sequences and integrating the model into a robotic grasping system, the method achieves a 98% success rate in object lifting with contact detection—a 52% improvement over open-loop grasps—demonstrating significant gains in robustness against external disturbances and grasp reliability.

ABSTRACT

In essence, successful grasp boils down to correct responses to multiple contact events between fingertips and objects. In most scenarios, tactile sensing is adequate to distinguish contact events. Due to the nature of high dimensionality of tactile information, classifying spatiotemporal tactile signals using conventional model-based methods is difficult. In this work, we propose to predict and classify tactile signal using deep learning methods, seeking to enhance the adaptability of the robotic grasp system to external event changes that may lead to grasping failure. We develop a deep learning framework and collect 6650 tactile image sequences with a vision-based tactile sensor, and the neural network is integrated into a contact-event-based robotic grasping system. In grasping experiments, we achieved 52% increase in terms of object lifting success rate with contact detection, significantly higher robustness under unexpected loads with slip prediction compared with open-loop grasps, demonstrating that integration of the proposed framework into robotic grasping system substantially improves picking success rate and capability to withstand external disturbances.

Motivation & Objective

• To address the challenge of interpreting high-dimensional spatiotemporal tactile signals for robotic grasping.
• To develop a data-driven approach that enables real-time detection and prediction of contact events, such as contact making, breaking, and slippage.
• To overcome limitations of model-based methods in generalizability and adaptability to diverse contact properties and external disturbances.
• To create a large-scale, human-annotated dataset of tactile sequences for training and evaluating contact event perception models.
• To integrate the proposed framework into a robotic grasping system to enhance robustness and success under unexpected loads.

Proposed method

• A deep learning framework, PixelMotionNet, is designed with two subnetworks: one for contact event classification and another for spatiotemporal tactile sequence prediction.
• The model processes raw tactile images and displacement vector fields from the FingerVision sensor, using HSV color space to encode direction, magnitude, and intensity of deformations.
• A human expert manually labels 6,650 tactile image sequences into seven explicit contact event categories based on neural firing patterns observed in human touch.
• The framework employs a joint training and evaluation scheme, with separate subnetworks trained for classification and video prediction tasks.
• The model is integrated into a Robotiq 2-finger gripper mounted on a UR10 robotic arm for real-time grasping experiments.
• Contact detection is used to trigger gripper adjustments only after contact is confirmed, enabling closed-loop grasping.

Experimental results

Research questions

• RQ1 Can a deep learning model effectively detect and predict multiple types of contact events from high-dimensional tactile sequences?
• RQ2 How does incorporating contact event prediction improve robotic grasp success under noisy or uncertain conditions?
• RQ3 To what extent does the proposed framework enhance robustness against external disturbances such as unexpected loads?
• RQ4 Can a data-driven approach outperform traditional model-based methods in classifying complex tactile interactions?
• RQ5 How effective is the model in detecting contact making and slip events when visual and proprioceptive modalities are absent?

Key findings

• The contact detection component increased the object lifting success rate from 46% (open-loop) to 98% across 10 diverse objects, representing a 52% improvement.
• With slip prediction, the grasping system maintained stable grasp under 7 loaded weights, whereas the open-loop system failed after only 3 weights due to slippage.
• The PixelMotionNet framework achieved high performance in both contact event classification and spatiotemporal sequence prediction, outperforming baseline models on the collected dataset.
• The human-annotated dataset of 6,650 tactile sequences provides a valuable benchmark for future research in tactile perception and robotic grasping.
• The model demonstrated strong generalization across different object shapes, sizes, and materials, indicating adaptability to real-world variations.
• The framework enables reactive grasping by detecting contact events and predicting slippage in real time, significantly enhancing system robustness.