[Paper Review] Measuring hand use in the home after cervical spinal cord injury using egocentric video

Andrea Bandini, Mehdy Dousty+4 more|arXiv (Cornell University)|Mar 31, 2022

Spinal Cord Injury Research43 references17 citations

TL;DR

This study developed and validated a wearable egocentric video system using deep learning to automatically measure hand use in individuals with tetraplegia at home. It demonstrated that automatically extracted metrics—percentage of interaction time (Perc) and number of interactions per hour (Num)—correlate significantly with clinical assessments of hand function and independence, establishing their validity as performance-based outcome measures in real-world settings.

ABSTRACT

Background: Egocentric video has recently emerged as a potential solution for monitoring hand function in individuals living with tetraplegia in the community, especially for its ability to detect functional use in the home environment. Objective: To develop and validate a wearable vision-based system for measuring hand use in the home among individuals living with tetraplegia. Methods: Several deep learning algorithms for detecting functional hand-object interactions were developed and compared. The most accurate algorithm was used to extract measures of hand function from 65 hours of unscripted video recorded at home by 20 participants with tetraplegia. These measures were: the percentage of interaction time over total recording time (Perc); the average duration of individual interactions (Dur); the number of interactions per hour (Num). To demonstrate the clinical validity of the technology, egocentric measures were correlated with validated clinical assessments of hand function and independence (Graded Redefined Assessment of Strength, Sensibility and Prehension - GRASSP, Upper Extremity Motor Score - UEMS, and Spinal Cord Independent Measure - SCIM). Results: Hand-object interactions were automatically detected with a median F1-score of 0.80 (0.67-0.87). Our results demonstrated that higher UEMS and better prehension were related to greater time spent interacting, whereas higher SCIM and better hand sensation resulted in a higher number of interactions performed during the egocentric video recordings. Conclusions: For the first time, measures of hand function automatically estimated in an unconstrained environment in individuals with tetraplegia have been validated against internationally accepted measures of hand function. Future work will necessitate a formal evaluation of the reliability and responsiveness of the egocentric-based performance measures for hand use.

Motivation & Objective

• To develop a wearable vision-based system for measuring hand function in individuals with tetraplegia in their natural home environment.
• To compare and select the most accurate deep learning algorithm for detecting functional hand-object interactions from egocentric video.
• To validate automatically extracted egocentric measures of hand use (Perc, Dur, Num) against established clinical assessments (GRASSP, UEMS, SCIM).
• To establish the clinical validity of vision-based metrics for capturing real-world performance of upper extremity function beyond clinical capacity measures.

Proposed method

• Employed egocentric video recordings from 20 participants with tetraplegia during unscripted daily activities at home.
• Evaluated multiple deep learning models for hand-object interaction detection, selecting the best-performing model based on F1-score.
• Extracted three key hand function metrics: Perc (percentage of interaction time), Dur (average interaction duration), and Num (interactions per hour).
• Correlated egocentric metrics with clinical assessments: GRASSP (prehension), UEMS (motor function), and SCIM (independence).
• Used a modular video processing pipeline with hand localization and interaction detection as core components.
• Applied computer vision techniques to infer functional hand use from hand-object contact states, with plans to extend to action and grasp recognition.

Experimental results

Research questions

• RQ1 Can deep learning accurately detect functional hand-object interactions in egocentric video recordings from individuals with tetraplegia in their home environment?
• RQ2 Do automatically extracted egocentric metrics (Perc, Dur, Num) correlate with validated clinical measures of hand function and independence?
• RQ3 Which clinical factors (e.g., motor function, sensation, prehension) are most strongly associated with specific egocentric hand use metrics?
• RQ4 Can egocentric video-based metrics serve as valid, performance-based outcome measures for upper extremity function in real-world settings?

Key findings

• The most accurate deep learning model achieved a median F1-score of 0.80 (IQR: 0.67–0.87) for detecting hand-object interactions.
• Higher UEMS scores were significantly correlated with greater Perc (r = 0.55, p = 0.01), indicating better motor function relates to more time spent interacting.
• Better prehension (GR-PP) was positively correlated with Perc (r = 0.48, p = 0.03), showing improved grasp quality increases interaction time.
• Higher SCIM scores and better hand sensation were associated with a greater number of interactions per hour (Num), suggesting improved independence and sensation support more frequent use.
• The study provides the first validation of egocentric vision-based hand use metrics against internationally accepted clinical assessments in individuals with tetraplegia.
• The modular processing pipeline enables future integration of grasp recognition and action detection to assess both quantity and quality of hand use.