[Paper Review] Weakly Supervised Segmentation with Multi-scale Adversarial Attention Gates

Gabriele Valvano, Andrea Leo+1 more|arXiv (Cornell University)|Jul 2, 2020

COVID-19 diagnosis using AI33 references3 citations

TL;DR

This paper proposes a weakly supervised segmentation model that leverages scribble annotations and a multi-scale generative adversarial network (GAN) with adversarial attention gates to generate high-quality segmentation masks. By conditioning attention gates on adversarial signals, the model learns shape priors, achieving performance on par with fully supervised models across medical and non-medical datasets.

ABSTRACT

Large, fine-grained image segmentation datasets, annotated at pixel-level, are difficult to obtain, particularly in medical imaging, where annotations also require expert knowledge. Weakly-supervised learning can train models by relying on weaker forms of annotation, such as scribbles. Here, we learn to segment using scribble annotations in an adversarial game. With unpaired segmentation masks, we train a multi-scale GAN to generate realistic segmentation masks at multiple resolutions, while we use scribbles to learn the correct position in the image. Central to the model's success is a novel attention gating mechanism, which we condition with adversarial signals to act as a shape prior, resulting in better object localization at multiple scales. We evaluated our model on several medical (ACDC, LVSC, CHAOS) and non-medical (PPSS) datasets, and we report performance levels matching those achieved by models trained with fully annotated segmentation masks. We also demonstrate extensions in a variety of settings: semi-supervised learning; combining multiple scribble sources (a crowdsourcing scenario) and multi-task learning (combining scribble and mask supervision). We will release expert-made scribble annotations for the ACDC dataset, and the code used for the experiments, at this https URL.

Motivation & Objective

• Address the challenge of acquiring large, pixel-level annotated medical images, which are costly and time-consuming to produce due to expert involvement.
• Develop a weakly supervised segmentation framework that uses only scribble-level annotations instead of full instance masks.
• Improve object localization and segmentation accuracy by incorporating adversarial signals into an attention gating mechanism as a shape prior.
• Demonstrate the model's effectiveness across diverse datasets, including medical (ACDC, LVSC, CHAOS) and non-medical (PPSS) domains.
• Extend the framework to semi-supervised, multi-source scribble, and multi-task learning settings to enhance robustness and generalization.

Proposed method

• Train a multi-scale GAN to generate realistic segmentation masks at multiple resolutions using unpaired ground-truth masks.
• Condition the generator with scribble annotations to guide object localization at the pixel level.
• Introduce a novel adversarial attention gate that uses feedback from the discriminator to refine feature maps and enforce shape consistency.
• Use adversarial signals to guide the attention gate, effectively acting as a shape prior that improves localization across scales.
• Train the generator and discriminator in an adversarial game, where the generator learns to produce realistic masks while the discriminator distinguishes real from fake masks.
• Integrate the attention gate into the skip connections of the U-Net-like architecture to preserve spatial details at all scales.

Experimental results

Research questions

• RQ1 Can a weakly supervised segmentation model using only scribble annotations achieve performance comparable to fully supervised models?
• RQ2 How effective is the proposed adversarial attention gate in improving object localization and segmentation accuracy?
• RQ3 Does the multi-scale GAN framework enhance the quality of generated segmentation masks across diverse image domains?
• RQ4 Can the model generalize to semi-supervised and multi-source scribble learning scenarios?
• RQ5 How does combining scribble supervision with partial mask supervision affect overall segmentation performance?

Key findings

• The proposed model achieves segmentation performance on par with fully supervised models on multiple medical and non-medical datasets, including ACDC, LVSC, CHAOS, and PPSS.
• The adversarial attention gate significantly improves object localization by acting as a shape prior, reducing false positives and enhancing boundary accuracy.
• The model generalizes well to semi-supervised learning, where only a subset of training samples are annotated with scribbles.
• Combining multiple scribble sources, such as from crowdsourced annotators, improves robustness and maintains high performance.
• The integration of scribble and partial mask supervision in a multi-task learning setup further enhances segmentation accuracy and convergence speed.
• The authors release expert-made scribble annotations for the ACDC dataset and code, supporting reproducibility and future research.