[Paper Review] A Multi-Stage GAN for Multi-Organ Chest X-ray Image Generation and Segmentation

Giorgio Ciano|arXiv (Cornell University)|Nov 14, 2021

Radiomics and Machine Learning in Medical Imaging50 references25 citations

TL;DR

This paper proposes a three-stage GAN framework for generating synthetic multi-organ chest X-ray images with corresponding semantic segmentation labels. By generating anatomical 'dots' first, then label-maps, and finally X-ray images through progressive refinement using PGGANs and pix2pixHD, the method achieves state-of-the-art segmentation performance even with only 10% of the training data, outperforming single- and two-stage baselines.

ABSTRACT

Multi-organ segmentation of X-ray images is of fundamental importance for computer aided diagnosis systems. However, the most advanced semantic segmentation methods rely on deep learning and require a huge amount of labeled images, which are rarely available due to both the high cost of human resources and the time required for labeling. In this paper, we present a novel multi-stage generation algorithm based on Generative Adversarial Networks (GANs) that can produce synthetic images along with their semantic labels and can be used for data augmentation. The main feature of the method is that, unlike other approaches, generation occurs in several stages, which simplifies the procedure and allows it to be used on very small datasets. The method was evaluated on the segmentation of chest radiographic images, showing promising results. The multi-stage approach achieves state-of-the-art and, when very few images are used to train the GANs, outperforms the corresponding single-stage approach.

Motivation & Objective

• Address the scarcity of labeled medical images for training deep learning models in chest X-ray segmentation.
• Overcome the challenge of limited annotated data in medical imaging by generating synthetic, high-quality training data.
• Develop a data augmentation strategy that preserves anatomical accuracy and improves segmentation performance.
• Demonstrate that a multi-stage generation process is more effective than single- or two-stage approaches, especially with small datasets.

Proposed method

• The method employs a three-stage GAN pipeline: first generating anatomical part locations as 'dots', then translating dots into semantic label-maps using pix2pixHD, and finally generating full X-ray images from label-maps.
• Progressively Growing GANs (PGGANs) are used in the initial stage to generate low-resolution anatomical dot maps.
• Pix2pixHD is applied in the second and third stages to perform conditional image-to-image translation from dots to label-maps and from label-maps to X-ray images.
• Each stage is trained independently, simplifying training and reducing data requirements per sub-task.
• The framework enables data augmentation by generating diverse, realistic synthetic images paired with accurate segmentation masks.
• The approach is modular and scalable, allowing extension to other anatomical structures or imaging modalities.

Experimental results

Research questions

• RQ1 Can a multi-stage GAN framework generate realistic, multi-organ chest X-ray images with accurate semantic segmentation masks from limited real data?
• RQ2 Does breaking down image generation into stages—dots → label-maps → images—improve performance compared to end-to-end generation?
• RQ3 How does the proposed method compare to single-stage and two-stage baselines in terms of segmentation accuracy when only 10% of the training data is used?
• RQ4 Can physicians reliably distinguish between ground-truth and generated segmentation masks, and does the generated output match clinical expectations?

Key findings

• The three-stage method achieved state-of-the-art performance on the SCR benchmark dataset for multi-organ chest X-ray segmentation.
• When trained on only 10% of the dataset, the three-stage approach outperformed both the two-stage and single-stage baselines in segmentation accuracy.
• Physician evaluations confirmed the high quality of generated segmentation masks, with experts frequently preferring the model’s output over the ground-truth in challenging cases.
• In qualitative assessments, physicians noted that the generated segmentations better preserved anatomical contours, particularly for the heart and lung margins.
• The method demonstrated robustness and clinical plausibility, even in cases with high segmentation difficulty, as evidenced by expert consensus on image quality.
• The multi-stage design significantly improved performance on small datasets, validating the hypothesis that stage-wise generation reduces complexity and enhances learning efficiency.