[Paper Review] Rapid-Motion-Track: Markerless Tracking of Fast Human Motion with Deeper Learning

Renjie Li, Chun Yu Lao+6 more|arXiv (Cornell University)|Jan 18, 2023

Stroke Rehabilitation and RecoveryMedicine3 citations

TL;DR

Rapid-Motion-Track (RMT) is a novel end-to-end deep learning system that enables accurate, markerless tracking of fast human movements—such as finger tapping at frequencies up to 8Hz—using standard 30fps laptop webcams. By leveraging a multi-scale feature fusion network (P-MSDSNet) and an Adaptive Vertex Recognition algorithm, RMT achieves 97.3% of movement features within ±0.5Hz of gold-standard wearable sensor data, significantly outperforming DeepLabCut (88.2% within ±0.5Hz).

ABSTRACT

Objective The coordination of human movement directly reflects function of the central nervous system. Small deficits in movement are often the first sign of an underlying neurological problem. The objective of this research is to develop a new end-to-end, deep learning-based system, Rapid-Motion-Track (RMT) that can track the fastest human movement accurately when webcams or laptop cameras are used. Materials and Methods We applied RMT to finger tapping, a well-validated test of motor control that is one of the most challenging human motions to track with computer vision due to the small keypoints of digits and the high velocities that are generated. We recorded 160 finger tapping assessments simultaneously with a standard 2D laptop camera (30 frames/sec) and a high-speed wearable sensor-based 3D motion tracking system (250 frames/sec). RMT and a range of DLC models were applied to the video data with tapping frequencies up to 8Hz to extract movement features. Results The movement features (e.g. speed, rhythm, variance) identified with the new RMT system exhibited very high concurrent validity with the gold-standard measurements (97.3\% of RMT measures were within +/-0.5Hz of the Optotrak measures), and outperformed DLC and other advanced computer vision tools (around 88.2\% of DLC measures were within +/-0.5Hz of the Optotrak measures). RMT also accurately tracked a range of other rapid human movements such as foot tapping, head turning and sit-to -stand movements. Conclusion: With the ubiquity of video technology in smart devices, the RMT method holds potential to transform access and accuracy of human movement assessment.

Motivation & Objective

• Develop a computer vision system capable of accurately tracking rapid human movements, such as finger tapping at frequencies above 4Hz, using standard low-frame-rate webcams.
• Overcome the limitations of existing deep learning models like DeepLabCut, which fail due to motion blur in low-frame-rate videos during high-velocity movements.
• Enable remote, scalable, and objective assessment of motor function in telemedicine and population-level screening by leveraging ubiquitous webcam technology.
• Provide a direct output of validated motor features (e.g., speed, rhythm, variance) rather than just keypoint coordinates, enhancing clinical and research utility.
• Extend the system’s applicability beyond finger tapping to other rapid movements such as foot tapping, head turning, and sit-to-stand transitions.

Proposed method

• Propose a novel deep learning architecture, P-MSDSNet, which fuses multi-scale features across both same and different depth levels in a parallel manner to improve keypoint detection on blurred frames.
• Implement an Adaptive Vertex Recognition algorithm to accurately identify peaks and troughs in distance-vs-time curves despite noise and unsmoothness, enabling robust feature extraction.
• Train and validate the RMT system end-to-end using synchronized video from standard 30fps laptop webcams and high-speed 250fps wearable Optotrak sensors.
• Use a dual-input pipeline: one for fingertip keypoint detection via P-MSDSNet, and another for feature extraction using adaptive vertex detection on the resulting time-series data.
• Compare RMT performance against multiple state-of-the-art models, including DeepLabCut variants (ResNet50 and MobileNet), using concurrent validation with gold-standard wearable sensor data.
• Design the system to output validated motor features directly, rather than only raw keypoint positions, enhancing usability for clinical and research applications.

Experimental results

Research questions

• RQ1 Can a deep learning-based system achieve high-accuracy markerless tracking of fast human movements (e.g., finger tapping >4Hz) using standard 30fps webcams?
• RQ2 How does RMT’s performance in tracking fingertip motion compare to state-of-the-art models like DeepLabCut under conditions of motion blur and high velocity?
• RQ3 To what extent can RMT extract valid and reliable motor features (e.g., speed, rhythm, variance) from low-frame-rate video without requiring high-speed cameras or wearable sensors?
• RQ4 Can RMT generalize to other rapid human movements such as foot tapping, head turning, and sit-to-stand transitions using the same framework?
• RQ5 Does RMT’s direct output of validated movement features improve clinical utility compared to systems that only provide keypoint coordinates?

Key findings

• RMT achieved 97.3% of movement features within ±0.5Hz of the gold-standard Optotrak wearable sensor system, demonstrating near-perfect concurrent validity.
• In contrast, DeepLabCut models (including ResNet50 and MobileNet variants) achieved only 88.2% of measures within ±0.5Hz of the Optotrak reference, indicating a significant performance gap in high-speed conditions.
• RMT successfully tracked finger tapping at frequencies up to 8Hz, a regime where motion blur renders existing computer vision models like DLC ineffective.
• The P-MSDSNet architecture effectively mitigates motion blur by fusing multi-scale features across depth levels, resulting in more stable and accurate fingertip tracking even on low-quality frames.
• The Adaptive Vertex Recognition algorithm enabled robust detection of peaks and troughs in distance-vs-time curves, ensuring accurate feature extraction despite noisy or unsmooth trajectories.
• RMT generalized effectively to other rapid movements, including head turning, toe tapping, leg agility, and sit-to-stand transitions, as validated by visual and quantitative analysis in supplementary materials.