[Paper Review] Generative Adversarial Networks for MR-CT Deformable Image Registration

Christine Tanner, Fırat Özdemir+4 more|arXiv (Cornell University)|Jul 19, 2018

Medical Image Segmentation Techniques5 references47 citations

TL;DR

The paper evaluates cycle-GAN-based MR-CT image synthesis for deformable registration and finds that synthesis can preserve or degrade spatial correspondences; abdomen registration can match state-of-the-art multi-modal methods, while thoracic results suffer due to lung volume biases.

ABSTRACT

Deformable Image Registration (DIR) of MR and CT images is one of the most challenging registration task, due to the inherent structural differences of the modalities and the missing dense ground truth. Recently cycle Generative Adversarial Networks (cycle-GANs) have been used to learn the intensity relationship between these 2 modalities for unpaired brain data. Yet its usefulness for DIR was not assessed. In this study we evaluate the DIR performance for thoracic and abdominal organs after synthesis by cycle-GAN. We show that geometric changes, which differentiate the two populations (e.g. inhale vs. exhale), are readily synthesized as well. This causes substantial problems for any application which relies on spatial correspondences being preserved between the real and the synthesized image (e.g. plan, segmentation, landmark propagation). To alleviate this problem, we investigated reducing the spatial information provided to the discriminator by decreasing the size of its receptive fields. Image synthesis was learned from 17 unpaired subjects per modality. Registration performance was evaluated with respect to manual segmentations of 11 structures for 3 subjects from the VISERAL challenge. State-of-the-art DIR methods based on Normalized Mutual Information (NMI), Modality Independent Neighborhood Descriptor (MIND) and their novel combination achieved a mean segmentation overlap ratio of 76.7, 67.7, 76.9%, respectively. This dropped to 69.1% or less when registering images synthesized by cycle-GAN based on local correlation, due to the poor performance on the thoracic region, where large lung volume changes were synthesized. Performance for the abdominal region was similar to that of CT-MRI NMI registration (77.4 vs. 78.8%) when using 3D synthesizing MRIs (12 slices) and medium sized receptive fields for the discriminator.

Motivation & Objective

• Motivate and assess MR-CT deformable image registration (DIR) when using cycle-GANs to synthesize one modality from the other.
• Investigate whether reducing discriminator receptive field mitigates geometric misalignment in synthesized images.
• Evaluate DIR performance with synthesized MR/CT against strong multi-modal DIR baselines (NMI, MIND, NMI+MIND) on thoracic and abdominal ROIs.
• Examine how region size and synthesis parameters affect registration outcomes.
• Provide guidance on balancing modality datasets to avoid cycle-GAN induced spatial biases.

Proposed method

• Adopt cycle-GANs to synthesize MR from CT and CT from MR using 2D residual-cycleGANs with PatchGAN discriminators.
• Train on 17 unpaired subjects per modality and evaluate 3D synthesis over thorax and abdomen ROIs.
• Incorporate cycle-consistency loss with a weighted objective that includes a cyclic L1 term (lambda_cyc = 10).
• Register CT/MR and synthesized images with a multi-modal DIR framework (ourDIR) that combines NMI and MIND dissimilarities.
• Use rigid initialization, then deformable registration with total variation or L2 regularization of the displacement field.
• Experiment with discriminator receptive field sizes (P×P patches) to study effects on geometric consistency.

Experimental results

Research questions

• RQ1 Can cycle-GAN synthesized MR/CT images enable accurate MR-CT deformable registration compared to standard multi-modal DIR measures?
• RQ2 How do region size and discriminator receptive field influence the geometric consistency and registration accuracy of synthesized images?
• RQ3 Do combinations of NMI and MIND dissimilarities improve DIR over single-modality or synthesis-based approaches?
• RQ4 What are the regional (thorax vs abdomen) differences in DIR performance when using synthesized images?

Key findings

• Registration with synthesized MR/CT can match or fall short of unpaired multi-modal methods depending on region; abdomen performance with synthesized data approaches CT-MR NMI results (~77.4–78.8% Dice range).
• Thoracic registrations show notable degradation with synthesized images due to lung-volume bias induced by cycle-GANs.
• Combining NMI and MIND (NMI+MIND) achieves competitive results for the thorax and abdomen, with best performance using the initial-gradient-based weighting (beta ≈ 0.8).
• 3D MR synthesis from CT and 3D CT synthesis from MR require careful region handling; inconsistent lung contours and slice-to-slice inconsistencies can occur, particularly with small ROI depths (C=3).
• Shallower discriminators (P=34) can reduce lung misalignment but may degrade synthesis quality, highlighting a trade-off between realism and geometric fidelity.