[Paper Review] DESK: A Robotic Activity Dataset for Dexterous Surgical Skills Transfer to Medical Robots

Naveen Madapana, M. K. Abd Rahman+11 more|arXiv (Cornell University)|Mar 3, 2019

Surgical Simulation and TrainingMedicine34 references3 citations

TL;DR

This paper introduces DESK, a multi-domain robotic surgical dataset comprising peg transfer tasks from a simulated Taurus II robot, a real Taurus II robot, and a YuMi industrial robot. It demonstrates that transfer learning from simulation data to real robotic systems significantly improves surgical gesture classification accuracy—achieving 55% accuracy with only simulated data and a 34% improvement when 3% real data is added—enabling skill transfer across diverse robotic platforms.

ABSTRACT

Datasets are an essential component for training effective machine learning models. In particular, surgical robotic datasets have been key to many advances in semi-autonomous surgeries, skill assessment, and training. Simulated surgical environments can enhance the data collection process by making it faster, simpler and cheaper than real systems. In addition, combining data from multiple robotic domains can provide rich and diverse training data for transfer learning algorithms. In this paper, we present the DESK (Dexterous Surgical Skill) dataset. It comprises a set of surgical robotic skills collected during a surgical training task using three robotic platforms: the Taurus II robot, Taurus II simulated robot, and the YuMi robot. This dataset was used to test the idea of transferring knowledge across different domains (e.g. from Taurus to YuMi robot) for a surgical gesture classification task with seven gestures. We explored three different scenarios: 1) No transfer, 2) Transfer from simulated Taurus to real Taurus and 3) Transfer from Simulated Taurus to the YuMi robot. We conducted extensive experiments with three supervised learning models and provided baselines in each of these scenarios. Results show that using simulation data during training enhances the performance on the real robot where limited real data is available. In particular, we obtained an accuracy of 55% on the real Taurus data using a model that is trained only on the simulator data. Furthermore, we achieved an accuracy improvement of 34% when 3% of the real data is added into the training process.

Motivation & Objective

• To address the lack of publicly available datasets for non-da Vinci surgical robots, which limits transfer learning across robotic platforms.
• To enable knowledge transfer from simulation to real-world robotic systems in minimally invasive surgery.
• To develop a benchmark dataset (DESK) that supports transfer learning across different robot morphologies and domains (simulated, real Taurus II, real YuMi).
• To evaluate the effectiveness of transfer learning in surgical gesture classification using limited real data and abundant simulation data.
• To provide a foundation for future research in domain adaptation, real-time surgeme detection, and cross-robot skill transfer in surgical robotics.

Proposed method

• The DESK dataset was collected using three platforms: a simulated Taurus II robot, a real Taurus II robot, and a real YuMi industrial robot.
• Surgical motions were recorded as synchronized RGB-D video and kinematic data during a peg transfer task.
• Surgeme instances were annotated using a semi-autonomous pipeline combining color-based image processing with human verification for bounding boxes and gesture boundaries.
• Sequence-wise and frame-wise features were extracted from the kinematic and visual data for classification tasks.
• Three supervised learning models (SVM, Random Forest, MLP) were trained and evaluated under three transfer learning scenarios: no transfer, simulated-to-real Taurus, and simulated-to-YuMi.
• Transfer learning was evaluated by training models on simulation data and fine-tuning with varying amounts of real robot data to measure performance gains.

Experimental results

Research questions

• RQ1 Can simulation-generated data effectively improve gesture classification accuracy on real robotic systems with limited real-world data?
• RQ2 How does transfer learning performance vary when transferring from a simulated robot to a physically different real robot (e.g., Taurus to YuMi)?
• RQ3 What is the impact of adding small amounts of real robot data to simulation-only training on classification accuracy?
• RQ4 How do sequence-wise versus frame-wise feature representations affect transfer learning performance in surgical gesture classification?
• RQ5 Can a model trained on simulation data detect surgeme boundaries in real-time without prior segmentation?

Key findings

• A model trained exclusively on simulated Taurus II data achieved a 55% accuracy in classifying surgical gestures on real Taurus II robot data, demonstrating effective zero-shot transfer from simulation to reality.
• Adding just 3% of real Taurus II data to the simulation training data improved classification accuracy by 34%, highlighting the value of small real-world data for fine-tuning.
• Transfer performance was lower when transferring from simulated Taurus to the YuMi robot (requiring α=0.3 for 80% accuracy) compared to Taurus (α=0.12), indicating greater domain shift between YuMi and Taurus.
• Sequence-wise features outperformed frame-wise features in transfer learning tasks, suggesting that temporal context is critical for accurate gesture recognition.
• Frame-wise classification showed promise for real-time application, as it does not require pre-segmented surgeme boundaries, enabling online detection of gesture start and end points.
• The dataset and code are publicly available at https://github.com/nmadapan/Forward_Project.git, supporting future research in cross-robot transfer learning and real-time surgical skill recognition.