[Paper Review] GANs for Medical Image Synthesis: An Empirical Study

Youssef Skandarani, Pierre‐Marc Jodoin+1 more|arXiv (Cornell University)|May 11, 2021

Generative Adversarial Networks and Image Synthesis28 citations

TL;DR

The paper evaluates multiple GAN architectures across three medical imaging modalities (cardiac cine-MRI, liver CT, and retinal images) using FID and downstream segmentation (Dice) to assess synthetic data usefulness, finding StyleGAN and SPADE excel in image fidelity but none reproduce full dataset richness; FID alone is not reliably predictive of segmentation performance.

ABSTRACT

Generative Adversarial Networks (GANs) have become increasingly powerful, generating mind-blowing photorealistic images that mimic the content of datasets they were trained to replicate. One recurrent theme in medical imaging is whether GANs can also be effective at generating workable medical data as they are for generating realistic RGB images. In this paper, we perform a multi-GAN and multi-application study to gauge the benefits of GANs in medical imaging. We tested various GAN architectures from basic DCGAN to more sophisticated style-based GANs on three medical imaging modalities and organs namely : cardiac cine-MRI, liver CT and RGB retina images. GANs were trained on well-known and widely utilized datasets from which their FID score were computed to measure the visual acuity of their generated images. We further tested their usefulness by measuring the segmentation accuracy of a U-Net trained on these generated images. Results reveal that GANs are far from being equal as some are ill-suited for medical imaging applications while others are much better off. The top-performing GANs are capable of generating realistic-looking medical images by FID standards that can fool trained experts in a visual Turing test and comply to some metrics. However, segmentation results suggests that no GAN is capable of reproducing the full richness of a medical datasets.

Motivation & Objective

• Assess how well various GAN architectures can synthesize realistic medical images across different organs and modalities.
• Determine if synthetic medical images can effectively augment training for downstream tasks like segmentation.
• Investigate the relationship between standard GAN fidelity metrics (FID) and downstream task performance.
• Identify practical considerations (data size, compute, stability) that affect medical image GAN utility.

Proposed method

• Select a range of GAN architectures (DCGAN, LSGAN, WGAN, HingeGAN, SPADE, StyleGAN) based on popularity and hardware compatibility.
• Train GANs on three datasets (ACDC cardiac cine-MRI, SLIVER07 liver CT, IDRiD retinal images) with extensive hyperparameter search (~500 GPU-days).
• Evaluate image fidelity with Frechet Inception Distance (FID) and assess downstream usefulness by training a U-Net segmentation model on synthetic data and measuring Dice score on real test sets.
• Concatenate image and mask channels for GANs to learn joint distributions, except SPADE which is conditioned on segmentation masks.
• Compare performance of augmented real data, synthetic data alone, and combinations thereof to assess practical utility.
• Discuss training tricks used to stabilize GANs (label smoothing, feature matching, differentiable augmentation).

Experimental results

Research questions

• RQ1 Do state-of-the-art and classic GANs produce medically plausible and diverse images across MRI, CT, and retinal datasets?
• RQ2 Can GAN-generated images meaningfully augment a downstream segmentation network trained on real data?
• RQ3 How do FID scores relate to task-specific Dice scores in medical imaging, and are there better domain-relevant metrics?
• RQ4 What practical factors (dataset size, 3D considerations, compute) influence the effectiveness of GAN-based synthetic data in medical imaging?

Key findings

• StyleGAN and SPADE achieve the best image fidelity and highest Dice scores across datasets, with StyleGAN reaching 87% Dice on ACDC and SPADE achieving 0.82–0.86 Dice depending on dataset.
• Simple GANs (DCGAN, LSGAN, WGAN, HingeGAN) perform poorly or variably, often yielding degenerate images and inconsistent segmentation performance.
• GAN-generated data rarely matches the segmentation performance obtained with real data, and augmentation with GAN data often matches traditional augmentations rather than surpasses them.
• FID correlates with human-judged visual quality but does not reliably predict downstream segmentation performance; some models with low FID memorized training data (e.g., SPADE on IDRiD with 1.09 FID and 82% Dice).
• More data (larger datasets like SLIVER07) generally yield better FID and downstream performance than very small datasets (IDRiD).
• High-quality medical generation often requires modern, compute-heavy architectures and may still be limited by the 2D nature of many GANs when data are inherently 3D.