[Paper Review] Toward Unpaired Multi-modal Medical Image Segmentation via Learning Structured Semantic Consistency

Jie Yang, Ye Zhu+3 more|arXiv (Cornell University)|Jun 21, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a novel unpaired multi-modal medical image segmentation method using a single Transformer backbone enhanced by a plug-and-play External Attention Module (EAM) to learn structured semantic consistency across modalities like CT and MRI. By aligning global class embeddings and image-level semantic correlations without requiring paired data, the method achieves state-of-the-art performance, improving mean Dice scores by 2.6% and 2.5% on two segmentation tasks, with the EAM removed at inference for model simplicity.

ABSTRACT

Integrating multi-modal data to promote medical image analysis has recently gained great attention. This paper presents a novel scheme to learn the mutual benefits of different modalities to achieve better segmentation results for unpaired multi-modal medical images. Our approach tackles two critical issues of this task from a practical perspective: (1) how to effectively learn the semantic consistencies of various modalities (e.g., CT and MRI), and (2) how to leverage the above consistencies to regularize the network learning while preserving its simplicity. To address (1), we leverage a carefully designed External Attention Module (EAM) to align semantic class representations and their correlations of different modalities. To solve (2), the proposed EAM is designed as an external plug-and-play one, which can be discarded once the model is optimized. We have demonstrated the effectiveness of the proposed method on two medical image segmentation scenarios: (1) cardiac structure segmentation, and (2) abdominal multi-organ segmentation. Extensive results show that the proposed method outperforms its counterparts by a wide margin.

Motivation & Objective

• Address the challenge of segmenting unpaired multi-modal medical images (e.g., CT and MRI) where no pixel-level correspondence exists.
• Overcome domain shift and modality discrepancy in multi-modal segmentation without requiring spatially aligned data.
• Learn structured semantic consistency—both class representation and inter-class correlation—across modalities to improve model generalization.
• Design a lightweight, modular component (EAM) that can be removed after training to preserve model simplicity and inference efficiency.
• Enable effective knowledge transfer between modalities with imbalanced data, where one modality has fewer samples than the other.

Proposed method

• Use a shared Transformer backbone to extract multi-scale features from unpaired CT and MRI images.
• Introduce modality-specific, learnable class embeddings per semantic class to represent global class-level features for each modality.
• Feed feature maps and class embeddings into an External Attention Module (EAM) to capture image-specific class representations and inter-class correlations at multiple scales.
• Enforce structured semantic consistency via consistency regularization at both modality-level and image-level, aligning class representations and their correlations across modalities.
• Train the model with EAMs active, then discard them during inference to maintain a lightweight, single-Transformer segmentation head.
• Leverage pretraining on ImageNet to stabilize training due to limited medical data availability.

Experimental results

Research questions

• RQ1 Can structured semantic consistency between unpaired multi-modal medical images (e.g., CT and MRI) be effectively learned without paired data or spatial alignment?
• RQ2 How can modality-specific and image-specific semantic representations be aligned to improve segmentation robustness across domain shifts?
• RQ3 Can a plug-and-play attention module enhance semantic consistency learning while allowing model simplification post-training?
• RQ4 To what extent can a low-resource modality improve the performance of a high-resource modality in unpaired settings?
• RQ5 How does the method generalize across different 2D and 3D medical image segmentation tasks with varying data distributions?

Key findings

• The proposed method achieves a 2.6% improvement in overall mean Dice score on cardiac structure segmentation compared to state-of-the-art baselines.
• On abdominal multi-organ segmentation, the method improves mean Dice by 2.5% over existing methods, demonstrating strong generalization.
• The model maintains high performance even under few-shot settings, where one modality has significantly fewer training samples than the other.
• A modality with fewer samples can still boost the training of a modality with more samples, indicating effective cross-modality knowledge transfer.
• The EAM module can be successfully removed after training, resulting in a lightweight inference model with a single Transformer backbone.
• Pretraining on ImageNet significantly improves performance, highlighting the importance of initialization in low-data medical segmentation settings.