[Paper Review] Cross-modality image synthesis from unpaired data using CycleGAN: Effects of gradient consistency loss and training data size

Yuta Hiasa, Yoshito Otake+6 more|arXiv (Cornell University)|Mar 18, 2018

Generative Adversarial Networks and Image Synthesis19 references21 citations

TL;DR

This paper proposes an enhanced CycleGAN framework with gradient consistency (GC) loss for MR-to-CT image synthesis in the musculoskeletal region, using a large unpaired dataset of 302 MR and 613 CT volumes. The GC loss improves boundary accuracy, and results show that larger training data and GC loss significantly boost both image synthesis fidelity and downstream segmentation performance, achieving a mean DICE of 0.808 ± 0.036 for pelvis and 0.883 ± 0.029 for femur on synthesized images.

ABSTRACT

CT is commonly used in orthopedic procedures. MRI is used along with CT to identify muscle structures and diagnose osteonecrosis due to its superior soft tissue contrast. However, MRI has poor contrast for bone structures. Clearly, it would be helpful if a corresponding CT were available, as bone boundaries are more clearly seen and CT has standardized (i.e., Hounsfield) units. Therefore, we aim at MR-to-CT synthesis. The CycleGAN was successfully applied to unpaired CT and MR images of the head, these images do not have as much variation of intensity pairs as do images in the pelvic region due to the presence of joints and muscles. In this paper, we extended the CycleGAN approach by adding the gradient consistency loss to improve the accuracy at the boundaries. We conducted two experiments. To evaluate image synthesis, we investigated dependency of image synthesis accuracy on 1) the number of training data and 2) the gradient consistency loss. To demonstrate the applicability of our method, we also investigated a segmentation accuracy on synthesized images.

Motivation & Objective

• To address the challenge of generating accurate CT images from MR scans in the hip region, where anatomical variability and lack of intensity consistency hinder prior methods.
• To improve boundary preservation in cross-modality synthesis by introducing gradient consistency loss into the CycleGAN framework.
• To evaluate the impact of training data size and GC loss on image synthesis accuracy and segmentation performance in musculoskeletal imaging.
• To demonstrate the feasibility of modality-independent segmentation using synthesized CT images from unlabeled MR data.

Proposed method

• Extended CycleGAN with gradient consistency (GC) loss to enforce structural alignment between real and synthesized images, particularly at tissue boundaries.
• Trained generators $G_{CT}$ and $G_{MR}$ to translate between MR and CT domains using cycle consistency and adversarial loss.
• Applied N4ITK intensity inhomogeneity correction to all MR volumes to reduce intensity non-uniformity.
• Used a 2D U-Net segmentation network with two-channel input (MR and synthesized CT) to perform joint segmentation on the synthesized data.
• Performed ablation studies by varying training data size (from 20 to 613 CT and 302 MR volumes) and including/excluding GC loss.
• Evaluated image synthesis using mutual information and qualitative visual inspection, and segmentation using DICE coefficient with paired ground truth.

Experimental results

Research questions

• RQ1 How does increasing the number of unpaired training data affect the quality of MR-to-CT image synthesis using CycleGAN?
• RQ2 Does incorporating gradient consistency loss improve boundary accuracy in synthesized CT images from MR input?
• RQ3 Can segmentation performance on MR volumes be significantly improved using synthesized CT images as auxiliary input?
• RQ4 How does the combination of large-scale data and GC loss compare to baseline CycleGAN in terms of segmentation accuracy and image fidelity?

Key findings

• Larger training data size (302 MR and 613 CT volumes) significantly improved image synthesis accuracy, as measured by mutual information, compared to smaller datasets.
• The inclusion of gradient consistency loss led to sharper boundaries in synthesized CT images, particularly around the femoral head and adductor muscles, as confirmed by visual inspection.
• Segmentation accuracy on synthesized images improved with both larger training data and GC loss, with a mean DICE of 0.808 ± 0.036 for pelvis and 0.883 ± 0.029 for femur.
• The GC loss specifically enhanced segmentation performance on the gluteus minimus (p < 0.01), achieving a DICE of 0.669 ± 0.054 with large data and GC.
• The modality-independent segmentation approach using synthesized CT achieved comparable or better results than single-modality MR segmentation, with DICE of 0.815 ± 0.046 for pelvis and 0.921 ± 0.023 for femur in 5-fold cross-validation.