[Paper Review] FingerVision Tactile Sensor Design and Slip Detection Using Convolutional LSTM Network

Yazhan Zhang, Zicheng Kan+3 more|arXiv (Cornell University)|Oct 5, 2018

Advanced Sensor and Energy Harvesting Materials19 references74 citations

TL;DR

The paper presents FingerVision, an optical-based tactile sensor with embedded markers, and a ConvLSTM-based framework for slip detection using deformation field data.

ABSTRACT

Tactile sensing is essential to the human perception system, so as to robot. In this paper, we develop a novel optical-based tactile sensor "FingerVision" with effective signal processing algorithms. This sensor is composed of soft skin with embedded marker array bonded to rigid frame, and a web camera with a fisheye lens. While being excited with contact force, the camera tracks the movements of markers and deformation field is obtained. Compared to existing tactile sensors, our sensor features compact footprint, high resolution, and ease of fabrication. Besides, utilizing the deformation field estimation, we propose a slip classification framework based on convolution Long Short Term Memory (convolutional LSTM) networks. The data collection process takes advantage of the human sense of slip, during which human hand holds 12 daily objects, interacts with sensor skin and labels data with a slip or non-slip identity based on human feeling of slip. Our slip classification framework performs high accuracy of 97.62% on the test dataset. It is expected to be capable of enhancing the stability of robot grasping significantly, leading to better contact force control, finer object interaction and more active sensing manipulation.

Motivation & Objective

• Develop a high-resolution, stable optical-based tactile sensor with easy fabrication.
• Capture dense deformation fields from marker movements for contact understanding.
• Detect slip behavior using a spatiotemporal learning model on deformation data.
• Evaluate slip detection performance against existing methods and analyze transferability.
• Demonstrate potential improvements for robust grasping and manipulation.

Proposed method

• Design of FingerVision: elastomer with embedded marker array, fisheye camera, and enclosed frame for stable marker tracking.
• Marker detection and tracking using blob detection and a frame-by-frame KNN correspondence approach.
• Deformation field reconstruction via RBF interpolation of tracked marker displacements.
• Convolutional LSTM network processing 10-frame multi-channel deformation field inputs (dx, dy, magnitude) to classify slip vs non-slip.
• Model details: ConvLSTM with 64 hidden units, 5x5 kernels, followed by a fully connected layer producing two-class output.
• Training with 800 slip and 800 non-slip samples (augmented to 8000 via sliding window), using cross-entropy loss and Adam optimizer.

Experimental results

Research questions

• RQ1 Can FingerVision provide reliable marker-based deformation fields for tactile sensing and contact characterization?
• RQ2 Can a ConvLSTM framework accurately detect slip events from spatiotemporal deformation data?
• RQ3 How does ConvLSTM-based slip detection compare to CNN+LSTM or univariate signal approaches in accuracy and efficiency?
• RQ4 Is the system suitable for real-time slip detection on robotic platforms?
• RQ5 How transferable is the approach across objects and fabrication variations?

Key findings

• Slip detection accuracy of 97.62% using interpolated deformation field with ConvLSTM.
• ConvLSTM model has 269,826 parameters and real-time forward pass time of 0.014 seconds on tested hardware.
• Compared methods achieve 88.03% (CNN+LSTM) and 94.5% (LSTM on spectral signals) with lower performance.
• FingerVision provides high-resolution, stable deformation fields suitable for dynamic grasping tasks.
• The approach demonstrates superior accuracy and potential for improved grasp stability and contact sensing in manipulation tasks.