[Paper Review] Finger Grip Force Estimation from Video using Two Stream Approach

Andrey Sartison, Dima Mironov+2 more|arXiv (Cornell University)|Mar 5, 2018

Hand Gesture Recognition Systems13 references3 citations

TL;DR

This paper proposes a noninvasive, two-stream video-based method to estimate finger grip force using spatial features from frames and temporal dynamics from optical flow, fused via a Kalman filter. The approach achieves an RMSE of approximately 0.2 N (less than 10% of the sensor range), enabling accurate force estimation without physical sensors, with applications in robot teaching and haptic feedback systems.

ABSTRACT

Estimation of a hand grip force is essential for the understanding of force pattern during the execution of assembly or disassembly operations. Human demonstration of a correct way of doing an operation is a powerful source of information which can be used for guided robot teaching. Typically to assess this problem instrumented approach is used, which requires hand or object mounted devices and poses an inconvenience for an operator or limits the scope of addressable objects. The work demonstrates that contact force may be estimated using a noninvasive contactless method with the help of vision system alone. We propose a two-stream approach for video processing, which utilizes both spatial information of each frame and dynamic information of frame change. In this work, image processing and machine learning techniques are used along with dense optical flow for frame change tracking and Kalman filter is used for stream fusion. Our studies show that the proposed method can successfully estimate contact grip force with RMSE < 10% of sensor range (RMSE $\approx 0.2$ N), the performances of each stream and overall method performance are reported. The proposed method has a wide range of applications, including robot teaching through demonstration, haptic force feedback, and validation of human- performed operations.

Motivation & Objective

• To develop a noninvasive method for estimating finger grip force without instrumented sensors or markers.
• To enable robot teaching through demonstration by extracting force profiles from video.
• To overcome limitations of invasive force sensors and marker-based systems in human-robot interaction and robotic disassembly.
• To leverage video data for creating annotated datasets of force and motion in human operations.
• To improve robustness and accuracy of force estimation using fused spatial and temporal video streams.

Proposed method

• A two-stream convolutional neural network processes spatial and temporal information from video frames.
• The spatial stream extracts static features from individual frames using image processing and skin segmentation.
• The temporal stream uses dense optical flow to capture motion dynamics and frame-to-frame changes.
• A Kalman filter fuses outputs from both streams to reduce noise and improve estimation stability.
• Signal filtering and image preprocessing enhance robustness to lighting and motion variations.
• Ground truth force data from FSR sensors is synchronized with video using a green LED trigger for training and evaluation.

Experimental results

Research questions

• RQ1 Can finger grip force be accurately estimated from video alone, without physical sensors or markers?
• RQ2 How do spatial and temporal video streams compare in performance for force estimation?
• RQ3 Can fusing spatial and temporal streams via Kalman filtering improve estimation accuracy and robustness?
• RQ4 How sensitive is the method to lighting changes, hand orientation, and slow force variations?
• RQ5 To what extent can this method be generalized across different individuals and object types?

Key findings

• The proposed two-stream method achieved an RMSE of approximately 0.2 N, representing less than 10% of the sensor range.
• The spatial stream showed consistent performance across force magnitudes, while the temporal stream struggled with slow force changes.
• Fusion via Kalman filter improved overall performance and reduced sensitivity to output magnitude.
• The temporal stream performed nearly as well as the full model, indicating that motion dynamics alone carry strong force-related information.
• The method demonstrated robustness to lighting variations, though performance was affected by hand movement and skin color response delays.
• The system is viable for robot teaching and force feedback in automated disassembly, especially for smartphones and similar electronics.