[Paper Review] DiamondGAN: Unified Multi-Modal Generative Adversarial Networks for MRI Sequences Synthesis

Hongwei Li, Johannes C. Paetzold+6 more|arXiv (Cornell University)|Apr 29, 2019

Generative Adversarial Networks and Image Synthesis14 references4 citations

TL;DR

DiamondGAN proposes a unified, scalable generative adversarial network for multi-modal MRI sequence synthesis, enabling flexible cross-modality translation from any subset of non-aligned input sequences (T1, T2, FLAIR) to target sequences (DIR, T1-c). It achieves radiologist-level indistinguishability in synthetic DIR images, with expert evaluations showing no significant difference between real and synthetic images (p = 0.1432).

ABSTRACT

Synthesizing MR imaging sequences is highly relevant in clinical practice, as single sequences are often missing or are of poor quality (e.g. due to motion). Naturally, the idea arises that a target modality would benefit from multi-modal input, as proprietary information of individual modalities can be synergistic. However, existing methods fail to scale up to multiple non-aligned imaging modalities, facing common drawbacks of complex imaging sequences. We propose a novel, scalable and multi-modal approach called DiamondGAN. Our model is capable of performing exible non-aligned cross-modality synthesis and data infill, when given multiple modalities or any of their arbitrary subsets, learning structured information in an end-to-end fashion. We synthesize two MRI sequences with clinical relevance (i.e., double inversion recovery (DIR) and contrast-enhanced T1 (T1-c)), reconstructed from three common sequences. In addition, we perform a multi-rater visual evaluation experiment and find that trained radiologists are unable to distinguish synthetic DIR images from real ones.

Motivation & Objective

• Address the clinical challenge of missing or poor-quality MRI sequences due to motion or protocol variation.
• Overcome limitations of existing GANs that require spatial alignment or are restricted to one-to-one or one-to-many mappings.
• Enable scalable, end-to-end learning of multiple-to-one cross-modality synthesis from arbitrary subsets of input modalities.
• Improve synthesis quality for clinically relevant sequences like double inversion recovery (DIR) and contrast-enhanced T1 (T1-c).
• Demonstrate clinical plausibility through multi-rater evaluation by experienced neuroradiologists.

Proposed method

• Introduce a condition vector c that encodes the presence of input modalities (e.g., [1,1,1] for T1, T2, FLAIR) and is spatially broadcast to match image dimensions.
• Design a two-stream generator architecture that takes both multi-modal input and the condition vector to produce target modality outputs.
• Employ a multi-modal cycle-consistency loss to enforce structural consistency across diverse input combinations, even without spatial alignment.
• Use paired discriminators (D1 and D2) to distinguish real vs. synthetic images from input modalities and target outputs, respectively.
• Train the model end-to-end using a unified framework that supports all possible input subsets, avoiding per-combination model training.
• Leverage a single generator and discriminator pair for all input combinations, enabling scalability and efficient inference.

Experimental results

Research questions

• RQ1 Can a unified GAN framework effectively synthesize high-quality MRI sequences from arbitrary subsets of non-aligned multi-modal inputs?
• RQ2 Does multi-modal input significantly improve synthesis quality for complex sequences like DIR and T1-c compared to single-modality inputs?
• RQ3 Can synthetic MRI sequences be indistinguishable from real ones in clinical evaluation by expert radiologists?
• RQ4 How does the proposed multi-modal cycle-consistency loss improve robustness and generalization across missing or variable input modalities?
• RQ5 To what extent does DiamondGAN outperform baseline models like CycleGAN in quantitative and qualitative metrics for MRI synthesis?

Key findings

• DiamondGAN achieved statistically significant improvements in PSNR and MAE over CycleGAN for DIR synthesis (p < 0.0001), with PSNR of 18.63 and MAE of 0.058 when using all three input modalities.
• Synthetic DIR images generated with T1+T2+Flair inputs were rated as 4.54 stars by radiologists, nearly matching the real DIR images (4.7 stars), with no significant difference (p = 0.1432).
• Radiologists could not distinguish synthetic DIR images from real ones when all three input modalities were used, indicating high clinical plausibility.
• The T1+T2+Flair combination yielded the highest PSNR (18.63) and lowest MAE (0.058) for DIR, and the highest PSNR (20.86) and lowest MAE (0.045) for T1-c among all configurations.
• Synthesis based solely on FLAIR was consistently rated as implausible by radiologists, highlighting the importance of multi-modal input for complex sequences.
• The model demonstrated robust performance across all input subsets, with no need for retraining per combination, confirming scalability and flexibility.