[Paper Review] Video and Accelerometer-Based Motion Analysis for Automated Surgical Skills Assessment

Aneeq Zia, Yachna Sharma+3 more|arXiv (Cornell University)|Feb 24, 2017

Surgical Simulation and Training4 citations

TL;DR

This paper proposes entropy-based features—Approximate Entropy (ApEn) and Cross-Approximate Entropy (XApEn)—to assess surgical skills using video and accelerometer data, outperforming existing methods like DCT, DFT, and SMT in automated OSATS evaluation. The fusion of video and accelerometer features with ApEn+XApEn achieves the highest accuracy, demonstrating superior performance on suturing and knot-tying tasks.

ABSTRACT

Purpose: Basic surgical skills of suturing and knot tying are an essential part of medical training. Having an automated system for surgical skills assessment could help save experts time and improve training efficiency. There have been some recent attempts at automated surgical skills assessment using either video analysis or acceleration data. In this paper, we present a novel approach for automated assessment of OSATS based surgical skills and provide an analysis of different features on multi-modal data (video and accelerometer data). Methods: We conduct the largest study, to the best of our knowledge, for basic surgical skills assessment on a dataset that contained video and accelerometer data for suturing and knot-tying tasks. We introduce "entropy based" features - Approximate Entropy (ApEn) and Cross-Approximate Entropy (XApEn), which quantify the amount of predictability and regularity of fluctuations in time-series data. The proposed features are compared to existing methods of Sequential Motion Texture (SMT), Discrete Cosine Transform (DCT) and Discrete Fourier Transform (DFT), for surgical skills assessment. Results: We report average performance of different features across all applicable OSATS criteria for suturing and knot tying tasks. Our analysis shows that the proposed entropy based features out-perform previous state-of-the-art methods using video data. For accelerometer data, our method performs better for suturing only. We also show that fusion of video and acceleration features can improve overall performance with the proposed entropy features achieving highest accuracy. Conclusions: Automated surgical skills assessment can be achieved with high accuracy using the proposed entropy features. Such a system can significantly improve the efficiency of surgical training in medical schools and teaching hospitals.

Motivation & Objective

• To develop a robust, automated system for surgical skills assessment to reduce reliance on expert manual evaluations.
• To address the limitations of existing methods that focus on motion repetitiveness while neglecting irregularity and predictability in surgical motions.
• To evaluate the performance of entropy-based features on multi-modal data (video and accelerometer) for OSATS-based surgical skill assessment.
• To compare the proposed entropy features with established methods like DCT, DFT, SMT, and motion texture techniques across a large-scale dataset.
• To investigate the effectiveness of early fusion between video and accelerometer features for improved assessment accuracy.

Proposed method

• Introduces Approximate Entropy (ApEn) and Cross-Approximate Entropy (XApEn) as novel features to quantify the regularity and predictability of time-series motion data from video and accelerometer sensors.
• Applies these entropy-based features to surgical motion sequences from suturing and knot-tying tasks to capture non-repetitive, irregular motion patterns indicative of skill level.
• Compares the entropy features against established methods: Sequential Motion Texture (SMT), Discrete Cosine Transform (DCT), and Discrete Fourier Transform (DFT).
• Employs an early fusion strategy by concatenating video and accelerometer features to leverage complementary information from both modalities.
• Uses 54 suturing and 62 knot-tying samples with synchronized video and accelerometer data, ensuring alignment across modalities.
• Performs cross-validation with 2-, 5-, and 10-fold schemes to evaluate robustness across varying training data sizes.

Experimental results

Research questions

• RQ1 Can entropy-based features (ApEn and XApEn) outperform traditional frequency and motion texture-based features in automated surgical skills assessment?
• RQ2 How does the performance of video-based features compare to accelerometer-based features for assessing OSATS criteria?
• RQ3 Does early fusion of video and accelerometer features improve overall classification accuracy for surgical skill assessment?
• RQ4 How robust are the proposed entropy-based features under varying training data sizes (as measured by cross-validation schemes)?
• RQ5 Do entropy features better capture skill-related motion irregularities compared to methods focused on repetitiveness (e.g., DFT, DCT) in surgical tasks?

Key findings

• The proposed ApEn+XApEn features achieved the highest average accuracy of 93.9% ± 3.7 on suturing and 94.0% ± 2.8 on knot tying when using fused video and accelerometer data.
• Video-only data outperformed accelerometer-only data across all feature types, with DCT achieving 90.6% ± 3.1 on suturing and 91.7% ± 6.1 on knot tying.
• Fusion of video and accelerometer data improved performance for ApEn+XApEn (93.2% on suturing, 94.0% on knot tying), while degrading performance for DCT and DFT.
• ApEn+XApEn features showed superior robustness across different cross-validation schemes (2-, 5-, 10-fold), consistently outperforming DCT and DFT.
• The entropy-based features demonstrated better generalization and sensitivity to motion irregularity, suggesting stronger suitability for diverse surgical tasks beyond repetitive ones.
• The study represents the largest known analysis of its kind, using 62 suturing and 54 knot-tying samples with synchronized multi-modal data.