[Paper Review] Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasets

Chinedu Innocent Nwoye, Nicolas Padoy|arXiv (Cornell University)|Apr 11, 2022

Surgical Simulation and Training19 references12 citations

TL;DR

This paper introduces standardized data splits and evaluation metrics for the CholecT45 and CholecT50 surgical action triplet datasets, enabling consistent benchmarking of deep learning models in surgical activity recognition. It proposes k-fold cross-validation splits for fair model comparison, develops the open-source ivtmetrics library for triplet evaluation, and releases reproducible models in PyTorch and TensorFlow, significantly improving comparability and progress tracking in the field.

ABSTRACT

In addition to generating data and annotations, devising sensible data splitting strategies and evaluation metrics is essential for the creation of a benchmark dataset. This practice ensures consensus on the usage of the data, homogeneous assessment, and uniform comparison of research methods on the dataset. This study focuses on CholecT50, which is a 50 video surgical dataset that formalizes surgical activities as triplets of . In this paper, we introduce the standard splits for the CholecT50 and CholecT45 datasets and show how they compare with existing use of the dataset. CholecT45 is the first public release of 45 videos of CholecT50 dataset. We also develop a metrics library, ivtmetrics, for model evaluation on surgical triplets. Furthermore, we conduct a benchmark study by reproducing baseline methods in the most predominantly used deep learning frameworks (PyTorch and TensorFlow) to evaluate them using the proposed data splits and metrics and release them publicly to support future research. The proposed data splits and evaluation metrics will enable global tracking of research progress on the dataset and facilitate optimal model selection for further deployment.

Motivation & Objective

• To establish consistent, reproducible data splits for the CholecT45 and CholecT50 datasets to enable fair comparison of surgical action triplet recognition methods.
• To define and standardize evaluation metrics for triplet detection and recognition, addressing inconsistencies in prior benchmarking practices.
• To develop and release a public, modular metrics library (ivtmetrics) for use across deep learning frameworks like PyTorch and TensorFlow.
• To re-implement and evaluate state-of-the-art models on the proposed splits, providing a reliable baseline for future research.
• To support long-term progress tracking and optimal model selection for clinical deployment by standardizing the evaluation protocol.

Proposed method

• Proposes two primary data splits: a standard train/val/test split aligned with prior publications and a k-fold cross-validation split to mitigate class imbalance and improve generalization assessment.
• Introduces a rigorous k-fold cross-validation strategy on CholecT45 and CholecT50 to ensure exhaustive evaluation across all data points and reduce variance in performance estimates.
• Designs and implements the ivtmetrics Python library for evaluating triplet recognition and localization, supporting both detection and classification metrics with standardization across frameworks.
• Re-implements existing state-of-the-art models for surgical triplet recognition in both PyTorch and TensorFlow to ensure reproducibility and cross-framework compatibility.
• Applies the standardized metrics and splits to evaluate models across multiple runs, ensuring consistent and comparable performance reporting.
• Releases trained models, code, weights, and the ivtmetrics library via GitHub and public datasets to promote transparency and reuse.

Experimental results

Research questions

• RQ1 What is the most effective and reproducible way to split the CholecT45 and CholecT50 datasets for benchmarking surgical action triplet recognition?
• RQ2 How can evaluation metrics be standardized to enable fair and consistent comparison of models across different research groups?
• RQ3 What performance can be achieved using state-of-the-art models when evaluated under standardized data splits and metrics?
• RQ4 How does k-fold cross-validation improve model evaluation reliability compared to single-test-set splits on this dataset?
• RQ5 To what extent does the proposed metrics library enable consistent and reusable evaluation across deep learning frameworks?

Key findings

• The proposed k-fold cross-validation split on CholecT45 and CholecT50 ensures more robust and generalizable model evaluation by reducing variance and mitigating class imbalance.
• The ivtmetrics library enables consistent, modular, and reusable evaluation of surgical triplet recognition models across PyTorch and TensorFlow frameworks.
• Reproduced models on the standardized splits show mean F1-scores of 29.4±2.8 on CholecT50 and 29.4±2.5 on CholecT45, providing a reliable baseline for future comparison.
• The benchmark study reveals that performance varies significantly across triplet classes, with high F1-scores (e.g., 94.0±4.6 for 'irrigator,irrigate,cystic-pedicle') and low scores (e.g., 0.5±0.1 for 'bipolar,grasp,cystic-plate'), highlighting class-specific challenges.
• The standard splits and metrics enable consistent tracking of research progress, with results now comparable across studies using the same protocol.
• The release of code, models, and metrics ensures reproducibility and accelerates future research in surgical action recognition.