[Paper Review] XCAT-GAN for Synthesizing 3D Consistent Labeled Cardiac MR Images on Anatomically Variable XCAT Phantoms

Sina Amirrajab, Samaneh Abbasi-Sureshjani+5 more|arXiv (Cornell University)|Jul 27, 2020

Generative Adversarial Networks and Image Synthesis28 references21 citations

TL;DR

This paper proposes XCAT-GAN, a conditional GAN framework that synthesizes 3D consistent, anatomically variable, labeled cardiac MR images using the 4D XCAT phantom. By training on real CMR images with semantic labels and transferring modality-specific textures to XCAT-derived label maps—using either 4-class (heart-only) or 8-class (multi-tissue) guidance—it generates high-fidelity synthetic data that improves cardiac cavity segmentation, retaining performance even when 80% of real data is replaced with synthetic images.

ABSTRACT

Generative adversarial networks (GANs) have provided promising data enrichment solutions by synthesizing high-fidelity images. However, generating large sets of labeled images with new anatomical variations remains unexplored. We propose a novel method for synthesizing cardiac magnetic resonance (CMR) images on a population of virtual subjects with a large anatomical variation, introduced using the 4D eXtended Cardiac and Torso (XCAT) computerized human phantom. We investigate two conditional image synthesis approaches grounded on a semantically-consistent mask-guided image generation technique: 4-class and 8-class XCAT-GANs. The 4-class technique relies on only the annotations of the heart; while the 8-class technique employs a predicted multi-tissue label map of the heart-surrounding organs and provides better guidance for our conditional image synthesis. For both techniques, we train our conditional XCAT-GAN with real images paired with corresponding labels and subsequently at the inference time, we substitute the labels with the XCAT derived ones. Therefore, the trained network accurately transfers the tissue-specific textures to the new label maps. By creating 33 virtual subjects of synthetic CMR images at the end-diastolic and end-systolic phases, we evaluate the usefulness of such data in the downstream cardiac cavity segmentation task under different augmentation strategies. Results demonstrate that even with only 20% of real images (40 volumes) seen during training, segmentation performance is retained with the addition of synthetic CMR images. Moreover, the improvement in utilizing synthetic images for augmenting the real data is evident through the reduction of Hausdorff distance up to 28% and an increase in the Dice score up to 5%, indicating a higher similarity to the ground truth in all dimensions.

Motivation & Objective

• Address the scarcity and limited anatomical variability in labeled cardiac MR datasets for deep learning.
• Overcome the limitation of existing domain translation methods that cannot generate new anatomical variations beyond the source domain.
• Develop a method to synthesize realistic, 3D-consistent, and semantically labeled CMR images using virtual phantoms with diverse anatomical features.
• Enable data augmentation and potential replacement of real data by generating synthetic images that preserve tissue-specific textures and anatomical consistency.
• Improve segmentation performance through synthetic data by incorporating multi-tissue label maps for better guidance in image synthesis.

Proposed method

• Utilize the 4D eXtended Cardiac and Torso (XCAT) phantom to generate 33 virtual subjects with diverse anatomical features, including heart and surrounding organ variations.
• Train a conditional GAN (XCAT-GAN) on real CMR images paired with semantic label maps to learn modality-specific image characteristics (e.g., T1/T2 contrast, tissue textures).
• Apply two synthesis strategies: 4-class (heart only) and 8-class (including multi-tissue labels for heart and surrounding organs) to improve spatial and semantic consistency.
• At inference, substitute real labels with XCAT-derived label maps to generate new synthetic CMR images while preserving the learned texture distribution.
• Leverage a pre-trained segmentation network on simulated XCAT images to generate accurate multi-tissue label maps for improved conditioning in 8-class synthesis.
• Use a 2D U-Net-based segmentation network for evaluation, trained on real data augmented with synthetic images to assess performance in cardiac cavity segmentation.

Experimental results

Research questions

• RQ1 Can XCAT-GAN generate 3D-consistent, labeled cardiac MR images with realistic tissue textures on anatomically variable XCAT phantoms?
• RQ2 Does incorporating multi-tissue label maps (8-class) improve the anatomical consistency and realism of synthesized CMR images compared to heart-only labels (4-class)?
• RQ3 To what extent can synthetic CMR images augment or replace real data in training deep learning models for cardiac cavity segmentation?
• RQ4 How does the inclusion of synthetic data affect segmentation performance in terms of Dice Score (DSC) and Hausdorff Distance (HD) under data-scarce conditions?
• RQ5 Can the synthetic data maintain high performance when real data is reduced to 20% of the original training set?

Key findings

• Synthetic CMR images generated by XCAT-GAN exhibit high anatomical and textural realism, with improved 3D consistency when using 8-class label maps.
• Segmentation performance is retained when training with only 20% of the real cCMR and ACDC data (40 volumes), provided synthetic data is included.
• The addition of synthetic data reduces the mean Hausdorff distance by up to 28% and increases the Dice score by up to 5% compared to baseline models.
• The 8-class synthesis leads to a significant reduction in false positives for background tissues, attributed to better anatomical guidance from multi-tissue labels.
• Performance improvements are consistent across multiple augmentation strategies, with the 8-class synthetic data showing superior generalization and robustness.
• The method enables replacement of up to 80% of real training data with synthetic images without significant performance drop, demonstrating strong potential for data-efficient deep learning in medical imaging.