[Paper Review] Synthetic data for unsupervised polyp segmentation

Enric Moreu, Kevin McGuinness+1 more|arXiv (Cornell University)|Feb 17, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes a fully unsupervised polyp segmentation method using synthetic data generated via 3D rendering and CycleGAN-based domain adaptation, achieving competitive performance on real-world datasets without any medical professional annotations. The key contribution is Synth-Colon, a 20,000-image synthetic dataset with depth and 3D mesh annotations, enabling state-of-the-art results even with limited real data.

ABSTRACT

Deep learning has shown excellent performance in analysing medical images. However, datasets are difficult to obtain due privacy issues, standardization problems, and lack of annotations. We address these problems by producing realistic synthetic images using a combination of 3D technologies and generative adversarial networks. We use zero annotations from medical professionals in our pipeline. Our fully unsupervised method achieves promising results on five real polyp segmentation datasets. As a part of this study we release Synth-Colon, an entirely synthetic dataset that includes 20000 realistic colon images and additional details about depth and 3D geometry: https://enric1994.github.io/synth-colon

Motivation & Objective

• Address the scarcity of annotated medical imaging data due to privacy, standardization, and expert annotation limitations in polyp segmentation.
• Develop a fully unsupervised pipeline that eliminates reliance on real-world medical annotations.
• Generate high-fidelity synthetic colonoscopy images with accurate polyp segmentation masks using 3D rendering and GAN-based domain adaptation.
• Demonstrate that synthetic data can achieve competitive performance on real polyp segmentation benchmarks, even outperforming some supervised methods in low-data regimes.
• Release Synth-Colon, the largest synthetic polyp segmentation dataset with depth and 3D geometry for public use.

Proposed method

• Generate synthetic colon and polyp structures using a 3D rendering engine with configurable anatomical and geometric parameters.
• Automatically generate ground-truth segmentation masks for polyps using the 3D engine’s internal geometry and spatial relationships.
• Apply CycleGAN to transfer the domain shift from synthetic to real colonoscopy images, using only real images from target datasets as real data.
• Train a HarDNeT-MSEG model on the realistic synthetic images and automatically generated segmentation masks, without any human-annotated labels.
• Optimize the CycleGAN using only real images from the target dataset, not combined datasets, to minimize domain gap and improve realism.
• Preserve self-generated segmentation annotations during domain adaptation by aligning synthetic and real image distributions via adversarial training.

Experimental results

Research questions

• RQ1 Can a polyp segmentation model be trained without any annotations from medical professionals using synthetic data?
• RQ2 How effective is domain adaptation via CycleGAN in transferring synthetic polyp images to real colonoscopy domains?
• RQ3 To what extent can synthetic data alone achieve performance comparable to fully supervised methods on real polyp segmentation benchmarks?
• RQ4 How does the performance of the unsupervised synthetic approach scale with limited real data availability?
• RQ5 Can synthetic data with depth and 3D mesh annotations improve downstream tasks like polyp segmentation and depth estimation?

Key findings

• The proposed unsupervised method achieved a mean Dice score of 0.703 on CVC-T, 0.521 on ColonDB, and 0.759 on Kvasir, demonstrating strong performance without any human-annotated labels.
• On the Kvasir dataset, the method achieved a Dice score of 0.642 with only five real images used for CycleGAN training, outperforming the supervised HarDNeT-MSEG model when fewer than 50 real images were available.
• The model trained on Synth-Colon outperformed several fully supervised baselines on the CVC-T and Kvasir datasets, indicating the effectiveness of synthetic data with domain adaptation.
• Training the CycleGAN on target-specific real images (rather than combined datasets) yielded better performance, highlighting the importance of domain-specific adaptation.
• Synth-Colon, the released dataset, includes 20,000 synthetic colon images with corresponding segmentation masks, depth maps, and 3D meshes in OBJ format, enabling broader use in medical imaging research.
• The method successfully preserved self-generated segmentation annotations during domain shift, enabling reliable training on realistic synthetic data without human labeling.