[Paper Review] On the effectiveness of GAN generated cardiac MRIs for segmentation

Youssef Skandarani, Nathan Painchaud+2 more|arXiv (Cornell University)|May 18, 2020

Advanced MRI Techniques and Applications8 references22 citations

TL;DR

This paper proposes a VAE-GAN framework that generates realistic, pixel-accurate cardiac MRIs with corresponding segmentation labels by conditioning on anatomical maps from a variational autoencoder. The method achieves state-of-the-art segmentation performance, improving Dice scores by up to 12% when combined with data augmentation and fine-tuning on limited real datasets.

ABSTRACT

In this work, we propose a Variational Autoencoder (VAE) - Generative Adversarial Networks (GAN) model that can produce highly realistic MRI together with its pixel accurate groundtruth for the application of cine-MR image cardiac segmentation. On one side of our model is a Variational Autoencoder (VAE) trained to learn the latent representations of cardiac shapes. On the other side is a GAN that uses "SPatially-Adaptive (DE)Normalization" (SPADE) modules to generate realistic MR images tailored to a given anatomical map. At test time, the sampling of the VAE latent space allows to generate an arbitrary large number of cardiac shapes, which are fed to the GAN that subsequently generates MR images whose cardiac structure fits that of the cardiac shapes. In other words, our system can generate a large volume of realistic yet labeled cardiac MR images. We show that segmentation with CNNs trained with our synthetic annotated images gets competitive results compared to traditional techniques. We also show that combining data augmentation with our GAN-generated images lead to an improvement in the Dice score of up to 12 percent while allowing for better generalization capabilities on other datasets.

Motivation & Objective

• To address the scarcity and high cost of annotated cardiac MRI datasets in medical imaging.
• To generate highly realistic, diverse, and fully supervised cardiac cine-MRI images with precise segmentation labels.
• To enable data augmentation for deep learning segmentation models using synthetic, anatomically valid MRI data.
• To improve generalization and performance of CNN-based segmentation networks on limited real datasets.
• To explore the feasibility of using VAE-GAN pipelines for end-to-end, label-consistent medical image generation.

Proposed method

• A variational autoencoder (VAE) learns the latent distribution of cardiac anatomical shapes from real segmentation maps.
• A GAN with SPADE (Spatially-Adaptive Normalization) modules generates realistic MR images conditioned on the VAE-generated anatomical maps.
• The generator produces MR images whose cardiac structures align precisely with the input anatomical maps, ensuring groundtruth consistency.
• During inference, sampling from the VAE's latent space produces diverse cardiac shapes, which are fed into the GAN to generate corresponding synthetic MRIs.
• The framework enables large-scale data generation (100k images) with paired image and segmentation labels for training.
• The method is evaluated by training ENet on synthetic data, with and without data augmentation and fine-tuning on real datasets.

Experimental results

Research questions

• RQ1 Can a VAE-GAN framework generate realistic cardiac MRIs with accurate, pixel-level segmentation labels?
• RQ2 How effective is the synthetic data from this framework in training CNNs for cardiac segmentation compared to real data?
• RQ3 Does combining synthetic data with traditional data augmentation improve segmentation performance and model generalization?
• RQ4 To what extent does fine-tuning on real data enhance the performance of models pre-trained on synthetic data?
• RQ5 Can the synthetic data improve cross-dataset generalization, especially when training on one dataset and testing on another?

Key findings

• The ENet trained on 100k synthetic ACDC-like images achieved a Dice score of 0.853 on the Sunnybrook dataset, compared to 0.773 when trained on real ACDC data, demonstrating superior cross-dataset generalization.
• With data augmentation and fine-tuning, the Dice score on the Sunnybrook dataset improved from 0.776 (real data only) to 0.874 (synthetic data with augmentation and fine-tuning).
• The model trained on synthetic data with data augmentation achieved a 12% relative improvement in Dice score compared to real data baseline, with gains of up to 4% over real data alone.
• The synthetic data outperformed real data in generalization, as shown by higher Dice scores on the target dataset when training on synthetic data.
• The VAE-GAN framework successfully generated diverse, anatomically valid cardiac shapes and corresponding realistic MRIs, with visual quality matching real cine-MRI.
• The combination of synthetic data and data augmentation led to consistent improvements across both ACDC and Sunnybrook datasets, with the best performance achieved via fine-tuning on real data after synthetic pre-training.