[Paper Review] Unsupervised Bidirectional Cross-Modality Adaptation via Deeply Synergistic Image and Feature Alignment for Medical Image Segmentation

Cheng Chen, Qi Dou+3 more|arXiv (Cornell University)|Feb 6, 2020

Domain Adaptation and Few-Shot Learning50 references10 citations

TL;DR

This paper proposes SIFA, a novel unsupervised domain adaptation framework for medical image segmentation that synergistically aligns cross-modality distributions by jointly optimizing image-to-image translation and feature-level domain invariance via adversarial learning with a shared encoder. Evaluated on MRI-CT cardiac and abdominal segmentation, SIFA significantly outperforms state-of-the-art methods, achieving higher Dice scores with bidirectional adaptation without target annotations.

ABSTRACT

Unsupervised domain adaptation has increasingly gained interest in medical image computing, aiming to tackle the performance degradation of deep neural networks when being deployed to unseen data with heterogeneous characteristics. In this work, we present a novel unsupervised domain adaptation framework, named as Synergistic Image and Feature Alignment (SIFA), to effectively adapt a segmentation network to an unlabeled target domain. Our proposed SIFA conducts synergistic alignment of domains from both image and feature perspectives. In particular, we simultaneously transform the appearance of images across domains and enhance domain-invariance of the extracted features by leveraging adversarial learning in multiple aspects and with a deeply supervised mechanism. The feature encoder is shared between both adaptive perspectives to leverage their mutual benefits via end-to-end learning. We have extensively evaluated our method with cardiac substructure segmentation and abdominal multi-organ segmentation for bidirectional cross-modality adaptation between MRI and CT images. Experimental results on two different tasks demonstrate that our SIFA method is effective in improving segmentation performance on unlabeled target images, and outperforms the state-of-the-art domain adaptation approaches by a large margin.

Motivation & Objective

• To address severe domain shift in medical image segmentation caused by differences in imaging modalities like MRI and CT.
• To improve unsupervised domain adaptation performance without requiring labeled target data.
• To unify image-level and feature-level domain alignment into a single end-to-end framework for synergistic improvement.
• To enable bidirectional adaptation between MRI and CT modalities, supporting both MRI-to-CT and CT-to-MRI transfer.
• To enhance model generalization across heterogeneous medical imaging data using adversarial learning and cycle-consistency.

Proposed method

• A shared feature encoder is used across both image and feature alignment branches to enable end-to-end training and mutual benefit.
• Image-level alignment is achieved via a CycleGAN-style generative adversarial network with cycle-consistency loss to translate source-domain images (e.g., MRI) to target-domain appearance (e.g., CT).
• Feature-level alignment is enforced using multiple discriminators that distinguish between real and fake features from source-like images and segmentation predictions.
• Deep supervision is applied by placing discriminators at multiple encoder stages to enhance feature-level domain invariance.
• The framework jointly optimizes image translation and segmentation via adversarial loss, cycle-consistency loss, and segmentation cross-entropy loss.
• The architecture supports bidirectional adaptation by alternating source and target roles in training, enabling zero-shot transfer between modalities.

Experimental results

Research questions

• RQ1 Can synergistic alignment of image and feature distributions improve unsupervised domain adaptation in cross-modality medical image segmentation?
• RQ2 How does joint optimization of image translation and feature alignment affect segmentation performance compared to isolated approaches?
• RQ3 To what extent can a shared encoder enhance the mutual benefits between image and feature adaptation in a unified framework?
• RQ4 Does the proposed method generalize across different anatomical regions and segmentation tasks (e.g., cardiac vs. abdominal organs)?
• RQ5 Can the framework achieve strong performance in bidirectional adaptation without any labeled target data?

Key findings

• SIFA achieves state-of-the-art performance on cardiac substructure segmentation, with an average Dice score of 90.9 on MRI and 83.6 on CT, surpassing supervised baselines in the target domain.
• On abdominal multi-organ segmentation, SIFA significantly improves segmentation accuracy over existing unsupervised domain adaptation methods, demonstrating robustness across complex anatomical structures.
• The method outperforms prior SOTA approaches by a large margin in both MRI-to-CT and CT-to-MRI adaptation, confirming its bidirectional effectiveness.
• Ablation studies show that removing either image or feature alignment leads to performance degradation, validating the synergistic contribution of both components.
• The framework maintains high performance even when the source and target domains have large distribution shifts, such as between MRI and CT scans.