[Paper Review] An Effective and Efficient Method for Detecting Hands in Egocentric Videos for Rehabilitation Applications

Ryan J. Visée, Jirapat Likitlersuang+1 more|arXiv (Cornell University)|Aug 27, 2019

Spinal Cord Injury ResearchMedicine43 references32 citations

TL;DR

This paper proposes a hybrid hand detection method combining YOLOv2 object detection with KCF or Median Flow tracking to achieve high accuracy and real-time performance in egocentric videos from individuals with spinal cord injury. The best combination (YOLO_KCF) achieved an F1-score of 0.87±0.07 while being twice as fast as the fastest detector alone, enabling efficient, robust hand tracking in home-based rehabilitation monitoring.

ABSTRACT

Objective: Individuals with spinal cord injury (SCI) report upper limb function as their top recovery priority. To accurately represent the true impact of new interventions on patient function and independence, evaluation should occur in a natural setting. Wearable cameras can be used to monitor hand function at home, using computer vision to automatically analyze the resulting videos (egocentric video). A key step in this process, hand detection, is difficult to do robustly and reliably, hindering deployment of a complete monitoring system in the home and community. We propose an accurate and efficient hand detection method that uses a simple combination of existing detection and tracking algorithms. Methods: Detection, tracking, and combination methods were evaluated on a new hand detection dataset, consisting of 167,622 frames of egocentric videos collected on 17 individuals with SCI performing activities of daily living in a home simulation laboratory. Results: The F1-scores for the best detector and tracker alone (SSD and Median Flow) were 0.90$\pm$0.07 and 0.42$\pm$0.18, respectively. The best combination method, in which a detector was used to initialize and reset a tracker, resulted in an F1-score of 0.87$\pm$0.07 while being two times faster than the fastest detector alone. Conclusion: The combination of the fastest detector and best tracker improved the accuracy over online trackers while improving the speed of detectors. Significance: The method proposed here, in combination with wearable cameras, will help clinicians directly measure hand function in a patient's daily life at home, enabling independence after SCI.

Motivation & Objective

• To develop a computationally efficient and accurate method for hand detection in egocentric videos from individuals with cervical spinal cord injury (SCI).
• To address the challenge of robust hand detection under real-world conditions such as occlusions, lighting variations, and camera motion.
• To enable real-time, in-home monitoring of hand function for upper limb rehabilitation using wearable cameras.
• To improve upon existing detection and tracking methods by combining their strengths into a lightweight, efficient system.
• To support clinical evaluation of new interventions by enabling direct measurement of hand function in daily living environments.

Proposed method

• The method combines YOLOv2 for initial hand detection with KCF or Median Flow tracking for continuous localization.
• A detector-only mode is used at initialization and after tracking failures to reinitialize the tracker.
• The system alternates between detection and tracking, avoiding simultaneous processing to reduce computational load.
• The approach uses a simple, non-parallel architecture that processes detection and tracking sequentially.
• The tracker is reset using the detector's output when tracking failure is detected, improving robustness to occlusions and fast motion.
• The method is evaluated on a new dataset of 167,622 frames from 17 individuals with SCI performing daily tasks in a home simulation lab.

Experimental results

Research questions

• RQ1 Can a hybrid detection-tracking system outperform standalone detectors or trackers in accuracy and speed for hand detection in egocentric videos?
• RQ2 How does the combination of YOLOv2 and KCF/median flow tracking perform under real-world challenges like occlusion and lighting variation?
• RQ3 Can the proposed method achieve real-time performance on low-power hardware suitable for portable, in-home rehabilitation systems?
• RQ4 How does the system’s robustness compare to standalone trackers, especially in handling tracking failures?
• RQ5 Does the method generalize to external datasets with different hand interaction patterns, such as social interactions?

Key findings

• The best standalone detector (SSD) achieved an F1-score of 0.90±0.07, while the best tracker (Median Flow) achieved 0.42±0.18.
• The YOLO_KCF combination achieved an F1-score of 0.87±0.07, outperforming the best tracker alone while being twice as fast as the fastest detector alone (68 FPS vs. 133 FPS).
• On a CPU-i5, the YOLO_KCF combination ran at 4.4 FPS, which is 15 times faster than YOLOv2 alone (0.3 FPS).
• On a CPU-i7, the YOLO_MF combination was estimated to achieve 20 FPS, meeting the target of 15–20 FPS for real-time deployment.
• The combination method showed lower standard deviation than trackers alone, indicating improved robustness and reliability.
• The method generalized well to external datasets: DAT achieved 0.722 average precision on EgoHands, outperforming prior work when considering only the camera-wearer’s hands.