[Paper Review] LC-GAN: Image-to-image Translation Based on Generative Adversarial Network for Endoscopic Images

Shan Lin, Fangbo Qin+4 more|arXiv (Cornell University)|Mar 10, 2020

Generative Adversarial Networks and Image Synthesis46 references4 citations

TL;DR

This paper proposes LC-GAN, a conditional image-to-image translation model based on generative adversarial networks to enable cross-domain surgical instrument segmentation using only a labeled cadaveric dataset and unlabeled live surgery images. By translating live images into fake-cadaveric images using two distinct generators with a pretrained backbone and introducing structural similarity and segmentation consistency losses, LC-GAN improves semantic fidelity and achieves 15.1–19.4% better mDSC and 17.9–22.9% better mIoU than CycleGAN, significantly reducing the need for manual labeling in live surgery segmentation.

ABSTRACT

Intelligent vision is appealing in computer-assisted and robotic surgeries. Vision-based analysis with deep learning usually requires large labeled datasets, but manual data labeling is expensive and time-consuming in medical problems. We investigate a novel cross-domain strategy to reduce the need for manual data labeling by proposing an image-to-image translation model live-cadaver GAN (LC-GAN) based on generative adversarial networks (GANs). We consider a situation when a labeled cadaveric surgery dataset is available while the task is instrument segmentation on an unlabeled live surgery dataset. We train LC-GAN to learn the mappings between the cadaveric and live images. For live image segmentation, we first translate the live images to fake-cadaveric images with LC-GAN and then perform segmentation on the fake-cadaveric images with models trained on the real cadaveric dataset. The proposed method fully makes use of the labeled cadaveric dataset for live image segmentation without the need to label the live dataset. LC-GAN has two generators with different architectures that leverage the deep feature representation learned from the cadaveric image based segmentation task. Moreover, we propose the structural similarity loss and segmentation consistency loss to improve the semantic consistency during translation. Our model achieves better image-to-image translation and leads to improved segmentation performance in the proposed cross-domain segmentation task.

Motivation & Objective

• To reduce the need for expensive manual labeling of live surgery endoscopic images in surgical instrument segmentation.
• To address domain shift between cadaveric and live endoscopic surgery images, where labeled data is available only in the cadaveric domain.
• To improve semantic consistency during image-to-image translation between endoscopic image domains using constrained GAN training.
• To enable transfer learning from a labeled cadaveric dataset to an unlabeled live surgery dataset via adversarial image translation.
• To demonstrate the feasibility of using image translation to augment training data and support surgical training and robotic assistance.

Proposed method

• LC-GAN employs two generators with different architectures: one uses a pretrained backbone (e.g., DeepLabV3+) to enhance feature representation from the cadaveric domain.
• The model is trained using a modified CycleGAN framework with unpaired real images from cadaveric and live surgery videos.
• A structural similarity loss ($\mathcal{L}_{SSim}$) is introduced to minimize structural distortions during image translation, preserving spatial and shape consistency.
• A segmentation consistency loss ($\mathcal{L}_{seg}$) is applied to align the segmentation predictions of real cadaveric images and their translated fake-cadaveric counterparts.
• The segmentation head trained on the real cadaveric dataset is applied to the translated fake-cadaveric images to perform inference on live surgery images.
• The overall loss function combines adversarial loss, cycle-consistency loss, structural similarity loss, and segmentation consistency loss for joint optimization.

Experimental results

Research questions

• RQ1 Can image-to-image translation between cadaveric and live endoscopic surgery images improve segmentation performance on unlabeled live surgery data?
• RQ2 How can structural and semantic consistency be preserved during translation between two visually distinct surgical image domains?
• RQ3 To what extent does incorporating a pretrained backbone and additional consistency losses improve translation and segmentation quality compared to standard GANs?
• RQ4 Can the proposed method reduce the need for manual labeling in live surgery instrument segmentation while maintaining high performance?
• RQ5 How does the performance of LC-GAN compare to baseline GANs like CycleGAN and other domain adaptation methods in cross-domain segmentation?

Key findings

• LC-GAN achieved 15.1% to 19.4% higher mean Dice Similarity Coefficient (mDSC) and 17.9% to 22.9% higher Intersection over Union (mIoU) than CycleGAN on the live surgery segmentation task.
• The ablation study confirmed that all three proposed components—backbone feature extractor, structural similarity loss, and segmentation consistency loss—significantly improved convergence and performance, with combined use yielding 0.2% to 2.2% further gains.
• Without any of the proposed modules, the network failed to converge within the training budget, highlighting the necessity of the introduced losses and architecture.
• LC-GAN outperformed UNIT and MUNIT, which struggled with domain shift due to their shared-latent-space assumption, especially in scenes with differing instruments and lighting.
• Despite lower performance than direct training on labeled live data (2.8–9.3% lower mDSC), LC-GAN’s cross-domain strategy provides a viable alternative when labeling live data is impractical.
• The method enables effective use of existing labeled cadaveric data for live surgery segmentation, reducing annotation burden while maintaining clinically relevant accuracy.