[Paper Review] Generative Adversarial Networks for Image-to-Image Translation on Multi-Contrast MR Images - A Comparison of CycleGAN and UNIT

Per Welander, Simon Karlsson+1 more|arXiv (Cornell University)|Jun 20, 2018

Generative Adversarial Networks and Image Synthesis10 references84 citations

TL;DR

The paper compares CycleGAN and UNIT (plus supervised variants) for translating between T1- and T2-weighted MR images using 2D slices, assessing quantitative metrics and perceived realism, and notes that visually realistic images do not always align with quantitative error measures.

ABSTRACT

In medical imaging, a general problem is that it is costly and time consuming to collect high quality data from healthy and diseased subjects. Generative adversarial networks (GANs) is a deep learning method that has been developed for synthesizing data. GANs can thereby be used to generate more realistic training data, to improve classification performance of machine learning algorithms. Another application of GANs is image-to-image translations, e.g. generating magnetic resonance (MR) images from computed tomography (CT) images, which can be used to obtain multimodal datasets from a single modality. Here, we evaluate two unsupervised GAN models (CycleGAN and UNIT) for image-to-image translation of T1- and T2-weighted MR images, by comparing generated synthetic MR images to ground truth images. We also evaluate two supervised models; a modification of CycleGAN and a pure generator model. A small perceptual study was also performed to evaluate how visually realistic the synthesized images are. It is shown that the implemented GAN models can synthesize visually realistic MR images (incorrectly labeled as real by a human). It is also shown that models producing more visually realistic synthetic images not necessarily have better quantitative error measurements, when compared to ground truth data. Code is available at https://github.com/simontomaskarlsson/GAN-MRI

Motivation & Objective

• Motivate the use of GANs to synthesize MR data and enable multimodal datasets for improved medical imaging tasks.
• Evaluate unsupervised CycleGAN and UNIT for image-to-image translation between T1- and T2-weighted MR images.
• Investigate supervised variants of CycleGAN and a generator-only model to understand the role of ground-truth guidance.
• Assess visual realism of synthetic MR images via a perceptual study.
• Provide public code to reproduce the experiments.

Proposed method

• Implement CycleGAN and UNIT in Keras for unpaired image-to-image translation between MR contrasts.
• Create two CycleGAN variants: CycleGAN_s (supervised with MAE) and Generators_s (generator-only with MAE).
• Include a simple two-layer CNN baseline as a control.
• Train on 2D axial slices from the Human Connectome Project dataset with 900 training and 213 testing images per domain.
• Evaluate with quantitative metrics (MAE, PSNR, MI) after normalizing images, and with a perceptual study to assess realism.
• Compare generation speed and training times per epoch (CycleGAN vs UNIT) and report qualitative results via visual differences.

Experimental results

Research questions

• RQ1 Which GAN model (CycleGAN vs UNIT) best translates MR images between T1 and T2 domains under unsupervised learning?
• RQ2 Do supervised training variants (CycleGAN_s, Generators_s) improve quantitative metrics, and at what cost to visual realism?
• RQ3 How do qualitative perceptual assessments of realism compare with quantitative error metrics across models?
• RQ4 What is the impact of using adversarial loss versus purely supervised MAE loss on image quality and edge detail?
• RQ5 Can these models provide a basis for data augmentation or multimodal MRI data synthesis?

Key findings

• The Generators_s model achieves the best performance across quantitative metrics (MAE, PSNR, MI).
• The Simple baseline model performs worst on quantitative metrics.
• CycleGAN, CycleGAN_s, and UNIT show similar quantitative performance, with T1 images generally scoring better than T2 in MAE.
• Perceptual study indicates synthetic MR images can be visually realistic (often labeled as real), with T2 images being harder to classify as synthetic.
• Supervised training (MAE) yields visually smooth, sometimes unrealistic images, suggesting adversarial loss is important for realism.
• Edges and CSF/white-matter boundaries present more challenges, particularly in T1 and T2 translations, respectively.