[Paper Review] Cross-Domain Medical Image Translation by Shared Latent Gaussian Mixture Model

Yingying Zhu, Youbao Tang+5 more|arXiv (Cornell University)|Jul 14, 2020

Generative Adversarial Networks and Image Synthesis19 references4 citations

TL;DR

This paper proposes a patch-based cross-domain medical image translation method using shared latent variables from a Gaussian mixture model to preserve fine anatomical structures during translation from non-contrast to contrast-enhanced CT. By modeling local image patches with a mixture of Gaussians instead of a single Gaussian, the method significantly improves preservation of small structures like calcified plaques and the pancreas, outperforming CycleGAN and UNIT in segmentation accuracy on both plaque and pancreas tasks.

ABSTRACT

Current deep learning based segmentation models often generalize poorly between domains due to insufficient training data. In real-world clinical applications, cross-domain image analysis tools are in high demand since medical images from different domains are often needed to achieve a precise diagnosis. An important example in radiology is generalizing from non-contrast CT to contrast enhanced CTs. Contrast enhanced CT scans at different phases are used to enhance certain pathologies or organs. Many existing cross-domain image-to-image translation models have been shown to improve cross-domain segmentation of large organs. However, such models lack the ability to preserve fine structures during the translation process, which is significant for many clinical applications, such as segmenting small calcified plaques in the aorta and pelvic arteries. In order to preserve fine structures during medical image translation, we propose a patch-based model using shared latent variables from a Gaussian mixture model. We compare our image translation framework to several state-of-the-art methods on cross-domain image translation and show our model does a better job preserving fine structures. The superior performance of our model is verified by performing two tasks with the translated images - detection and segmentation of aortic plaques and pancreas segmentation. We expect the utility of our framework will extend to other problems beyond segmentation due to the improved quality of the generated images and enhanced ability to preserve small structures.

Motivation & Objective

• To address the poor generalization of deep learning segmentation models across medical imaging domains due to limited labeled data.
• To improve the preservation of fine anatomical structures—such as calcified plaques and small organs—during cross-domain image translation.
• To develop a method that works with unpaired images, enabling practical application in real-world clinical settings where paired data are rare.
• To enhance segmentation performance on non-contrast CT by generating high-fidelity synthetic contrast-enhanced CT images.

Proposed method

• The method uses a patch-based approach, extracting small 32×32 image patches from non-contrast and contrast-enhanced CT scans to model local image statistics.
• It models the shared latent space using a Gaussian Mixture Model (GMM) instead of a single Gaussian distribution, capturing multiple local clusters in image patches.
• The GMM-based latent space is learned via variational autoencoders with shared components across domains, enabling domain-invariant representation learning.
• Image translation is performed by mapping patches from one domain to another using the shared GMM-structured latent space.
• The model is trained end-to-end using adversarial loss and cycle-consistency loss to ensure realistic and consistent translations.
• The framework is applied to generate synthetic non-contrast CT from contrast-enhanced CT, which are then used to augment training data for segmentation tasks.

Experimental results

Research questions

• RQ1 Can a shared latent space modeled by a Gaussian Mixture Model preserve fine anatomical structures better than a single Gaussian distribution in cross-domain medical image translation?
• RQ2 Does the proposed patch-based GMM model improve segmentation performance for small structures like calcified plaques in aortic and pelvic arteries?
• RQ3 How does the GMM-based translation method compare to state-of-the-art models like CycleGAN and UNIT in preserving structural details during CT image translation?
• RQ4 To what extent can synthetic images generated by the proposed method improve segmentation accuracy on non-contrast CT scans when used for training?
• RQ5 Can the method generalize to other segmentation tasks, such as pancreas segmentation, in low-contrast or non-contrast imaging domains?

Key findings

• The proposed method achieved a Dice score of 0.734 ± 0.136 for aortic plaque segmentation on non-contrast CT, outperforming CycleGAN (0.728 ± 0.154) and UNIT (0.731 ± 0.142).
• For pancreas segmentation, the model improved Dice score by over 0.08 compared to the baseline when training on synthetic non-contrast CT, with a reduction of over 5,000 false positive pixels.
• The method reduced the drop in precision and recall for plaque segmentation on non-contrast CT by more than 1.5% and 4%, respectively, compared to the baseline.
• Segmentation performance on contrast-enhanced CT was also improved when synthetic non-contrast images were used for training, with the proposed method showing the highest Dice score among all methods.
• The model demonstrated superior preservation of fine structures, as visually confirmed by reduced blurring of calcified plaques compared to CycleGAN and UNIT.
• The method achieved state-of-the-art performance in both plaque and pancreas segmentation tasks, validating its ability to generate high-quality synthetic images for cross-domain medical image analysis.