[Paper Review] Weakly-Supervised Learning for Tool Localization in Laparoscopic Videos

Armine Vardazaryan, Didier Mutter+2 more|arXiv (Cornell University)|Jun 14, 2018

Surgical Simulation and Training17 references4 citations

TL;DR

This paper proposes a weakly-supervised deep learning method for surgical tool localization in laparoscopic videos using only image-level annotations, avoiding costly spatial annotations. By adapting a fully convolutional network (FCN) with ResNet18 backbone and extended spatial pooling (ESP), the model generates localization heatmaps that predict tool presence and positions with high accuracy, achieving 88% mean average precision on the Cholec80 dataset.

ABSTRACT

Surgical tool localization is an essential task for the automatic analysis of endoscopic videos. In the literature, existing methods for tool localization, tracking and segmentation require training data that is fully annotated, thereby limiting the size of the datasets that can be used and the generalization of the approaches. In this work, we propose to circumvent the lack of annotated data with weak supervision. We propose a deep architecture, trained solely on image level annotations, that can be used for both tool presence detection and localization in surgical videos. Our architecture relies on a fully convolutional neural network, trained end-to-end, enabling us to localize surgical tools without explicit spatial annotations. We demonstrate the benefits of our approach on a large public dataset, Cholec80, which is fully annotated with binary tool presence information and of which 5 videos have been fully annotated with bounding boxes and tool centers for the evaluation.

Motivation & Objective

• Address the challenge of limited spatial annotations in surgical video datasets, which restricts model scalability and generalization.
• Enable tool localization without requiring bounding boxes or segmentation masks during training.
• Leverage the feature hierarchy of deep networks to infer spatial positions from image-level labels alone.
• Demonstrate the feasibility of weakly-supervised learning for surgical tool detection and localization on large-scale public datasets like Cholec80.
• Provide a foundation for automated annotation tools and future segmentation methods in surgical video analysis.

Proposed method

• Adapt a ResNet18 backbone by replacing the final fully connected layer and average pooling with a fully convolutional architecture to preserve spatial resolution.
• Reduce the stride in the last two residual blocks from 2 to 1 to increase feature map resolution, quadrupling output resolution with a global stride of 8.
• Use a 1×1 convolutional layer to produce C localization heatmaps (C=7 for tools), followed by extended spatial pooling (ESP) to enhance object detection details.
• Apply ESP pooling: $ s^c = \max z^c + \alpha \min z^c $, with $\alpha = 0.6$, to generate class-wise confidence scores and refine localization maps.
• During inference, upsample heatmaps via bilinear interpolation and predict tool centers as the location of maximum activation.
• Introduce data augmentation techniques including random cropping, color jittering, and mixup to improve robustness and generalization.

Experimental results

Research questions

• RQ1 Can a weakly-supervised deep learning model achieve accurate surgical tool localization using only image-level annotations?
• RQ2 How does the use of extended spatial pooling (ESP) compare to standard pooling in improving localization precision?
• RQ3 To what extent do data augmentation strategies like mixup and masking improve model generalization on surgical video data?
• RQ4 How does the model perform on tools with ambiguous shapes or variable appearances, such as the specimen bag?
• RQ5 Can the learned localization maps be used to guide future weakly-supervised segmentation or automated annotation pipelines?

Key findings

• The proposed weakly-supervised method achieves a mean average precision (mAP) of 88% for tool localization on the Cholec80 test set, demonstrating strong performance without spatial annotations.
• Models using both ESP and masking (FCN_ESP_Msk) achieved the lowest mean distance error (6.8%) between predicted and ground-truth tool centers, outperforming other variants.
• The specimen bag was localized with the highest error (9.7% mean distance), attributed to its variable shape and ambiguous center point.
• Localization maps from models with ESP and masking showed better coverage of tool regions and more discriminative activation patterns, especially for tools like scissors and irrigators.
• The model detected the shafts of tools like scissors and irrigators as highly discriminative features, which may explain lower AP scores due to mismatch with ground-truth bounding boxes focused on tool tips.
• Qualitative results confirmed that FCN_ESP_MM_Msk produced the most consistent and detailed localization maps, with strong activation at tool tips and minimal false positives.