[Paper Review] Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network

Zizhao Zhang, Lin Yang+1 more|arXiv (Cornell University)|Feb 27, 2018

Generative Adversarial Networks and Image SynthesisComputer Science42 references53 citations

TL;DR

Proposes a 3D end-to-end GAN framework that jointly translates cross-modality medical volumes (CT/MRI) and improves segmentation by incorporating cycle- and shape-consistency losses and online synthetic data augmentation.

ABSTRACT

Synthesized medical images have several important applications, e.g., as an intermedium in cross-modality image registration and as supplementary training samples to boost the generalization capability of a classifier. Especially, synthesized computed tomography (CT) data can provide X-ray attenuation map for radiation therapy planning. In this work, we propose a generic cross-modality synthesis approach with the following targets: 1) synthesizing realistic looking 3D images using unpaired training data, 2) ensuring consistent anatomical structures, which could be changed by geometric distortion in cross-modality synthesis and 3) improving volume segmentation by using synthetic data for modalities with limited training samples. We show that these goals can be achieved with an end-to-end 3D convolutional neural network (CNN) composed of mutually-beneficial generators and segmentors for image synthesis and segmentation tasks. The generators are trained with an adversarial loss, a cycle-consistency loss, and also a shape-consistency loss, which is supervised by segmentors, to reduce the geometric distortion. From the segmentation view, the segmentors are boosted by synthetic data from generators in an online manner. Generators and segmentors prompt each other alternatively in an end-to-end training fashion. With extensive experiments on a dataset including a total of 4,496 CT and magnetic resonance imaging (MRI) cardiovascular volumes, we show both tasks are beneficial to each other and coupling these two tasks results in better performance than solving them exclusively.

Motivation & Objective

• Address the lack of paired cross-modality medical volumes for translation between CT and MRI.
• Develop a 3D GAN framework that preserves anatomical shape during translation via shape-consistency.
• Jointly train generators and segmentors to exploit synthetic data for improved segmentation performance.
• Demonstrate that online use of synthetic data outperforms offline augmentation in both translation quality and segmentation.
• Validate the approach on a large cardiovascular CT/MRI dataset to show mutual benefits of translation and segmentation.

Proposed method

• Use two generators G_A and G_B for cross-domain volume translation with discriminators D_A and D_B.
• Apply cycle-consistency loss to enforce G_A(G_B(x_A)) ≈ x_A and G_B(G_A(x_B)) ≈ x_B.
• Introduce shape-consistency loss via segmentors S_A and S_B to map translated data to a shared shape space Y and compute cross-entropy losses.
• Train segmentors S_A and S_B with both real and online synthetic data to boost segmentation, including reconstructed synthetic data.
• Combine adversarial, cycle-consistency, and shape-consistency losses into a joint objective L = L_GAN + L_GAN + λ L_cyc + γ L_shape.
• Provide end-to-end training where generators and segmentors mutually reinforce each other.

Experimental results

Research questions

• RQ1 Can unpaired CT and MRI data be translated across modalities while preserving anatomical shapes in 3D volumes?
• RQ2 Does incorporating a shape-consistency constraint improve cross-modality translation quality over cycle-consistency alone?
• RQ3 Can synthetic data generated in an online, coupled training framework improve segmentation performance more than offline augmentation?
• RQ4 What is the impact of jointly training generators and segmentors on translation quality and segmentation accuracy?

Key findings

• Shape-consistency improves translation quality, yielding higher shape-quality scores (S-score) than without shape-consistency.
• End-to-end online use of synthetic data substantially improves segmentation performance over baseline and offline augmentation.
• On a dataset of 4,496 cardiovascular CT/MRI volumes, the method achieves higher Dice scores for CT and MRI segmentation when using synthetic data (74.4% CT, 73.2% MRI) compared to baseline and ADA baselines.
• Shape-consistency-enabled generator models produce synthetic volumes with fewer artifacts and better anatomical preservation than CycleGAN baselines.
• Jointly training generators and segmentors yields better overall results than solving translation and segmentation separately.