[Paper Review] Synergistic Image and Feature Adaptation: Towards Cross-Modality Domain Adaptation for Medical Image Segmentation

Cheng Chen, Qi Dou+3 more|arXiv (Cornell University)|Jan 24, 2019

Domain Adaptation and Few-Shot LearningComputer Science25 references42 citations

TL;DR

Introduces SIFA, an unsupervised cross-modality domain adaptation framework that synergistically combines image appearance transformation with feature-invariant learning to improve medical image segmentation without target-domain annotations.

ABSTRACT

This paper presents a novel unsupervised domain adaptation framework, called Synergistic Image and Feature Adaptation (SIFA), to effectively tackle the problem of domain shift. Domain adaptation has become an important and hot topic in recent studies on deep learning, aiming to recover performance degradation when applying the neural networks to new testing domains. Our proposed SIFA is an elegant learning diagram which presents synergistic fusion of adaptations from both image and feature perspectives. In particular, we simultaneously transform the appearance of images across domains and enhance domain-invariance of the extracted features towards the segmentation task. The feature encoder layers are shared by both perspectives to grasp their mutual benefits during the end-to-end learning procedure. Without using any annotation from the target domain, the learning of our unified model is guided by adversarial losses, with multiple discriminators employed from various aspects. We have extensively validated our method with a challenging application of cross-modality medical image segmentation of cardiac structures. Experimental results demonstrate that our SIFA model recovers the degraded performance from 17.2% to 73.0%, and outperforms the state-of-the-art methods by a significant margin.

Motivation & Objective

• Address the severe domain shift between medical imaging modalities (e.g., MR vs CT) in segmentation tasks.
• Develop an unsupervised framework that combines image appearance transformation with feature-level domain alignment.
• Share representations across image and feature adaptations to enable end-to-end, zero-annotation transfer.
• Demonstrate effectiveness on cross-modality cardiac structure segmentation and beat prior methods on Dice/ ASD metrics.

Proposed method

• Use image adaptation to transform source domain images into target-like appearances via a generative adversarial network with cycle-consistency.
• Jointly train a feature adapter by introducing discriminators in compact spaces: semantic prediction space and generated image space.
• Share the encoder between image and feature adaptation to learn unified, domain-invariant representations.
• Optimize with a hybrid segmentation loss (cross-entropy + Dice) to handle class imbalance in medical data.
• Leverage three discriminators (D_t, D_s, D_p) and two generators (G_t, G_s) within an end-to-end adversarial learning framework.
• Train in an alternating sequence that updates image translation, discriminators, encoder, classifier, decoder, and feature-space discriminators.

Experimental results

Research questions

• RQ1 Can synergistic, end-to-end image and feature adaptations reduce cross-modality domain gap for medical image segmentation without target annotations?
• RQ2 Does sharing the feature encoder enable mutual benefits between image appearance transformation and feature-invariance learning?
• RQ3 How much do compact-space discriminators (semantic and generated-image spaces) improve domain-invariant feature learning beyond image-only or feature-only approaches?
• RQ4 Is the framework effective for challenging cross-modality tasks such as MR-to-CT cardiac structure segmentation?

Key findings

• SIFA recovers segmentation performance from 17.2% Dice (no adaptation) to 73.0% average Dice across four cardiac structures on MR to CT segmentation.
• SIFA outperforms state-of-the-art unsupervised domain adaptation methods across Dice and ASD metrics on the cardiac segmentation task.
• Image adaptation alone provides substantial gains; adding feature adaptation in semantic prediction space further improves performance, and adding generated-source space adaptation yields the best results.
• Compared to CyCADA, SIFA achieves superior performance on LVC and MYO structures, indicating benefits of the synergistic, bidirectional adaptation framework.
• Ablation shows that combining image adaptation with both feature-adaptation components yields the strongest Dice score improvement (average Dice 73.0).