[Paper Review] Learning Cross-Modal Deep Representations for Multi-Modal MR Image Segmentation

Cheng Li, Hui Sun+4 more|arXiv (Cornell University)|Aug 6, 2019

Advanced Neural Network Applications13 references4 citations

TL;DR

This paper proposes a supervised cross-modal feature fusion method for multi-modal MR image segmentation using attention-guided, master-modality-driven information selection. By leveraging a master modality (e.g., T1-enhanced MRI) to guide attention-based fusion of auxiliary modalities (e.g., T2-weighted MRI), the method improves segmentation accuracy by selectively emphasizing relevant features and suppressing noise, achieving state-of-the-art performance on breast mass segmentation with a Dice score of 84.4±0.7.

ABSTRACT

Multi-modal magnetic resonance imaging (MRI) is essential in clinics for comprehensive diagnosis and surgical planning. Nevertheless, the segmentation of multi-modal MR images tends to be time-consuming and challenging. Convolutional neural network (CNN)-based multi-modal MR image analysis commonly proceeds with multiple down-sampling streams fused at one or several layers. Although inspiring performance has been achieved, the feature fusion is usually conducted through simple summation or concatenation without optimization. In this work, we propose a supervised image fusion method to selectively fuse the useful information from different modalities and suppress the respective noise signals. Specifically, an attention block is introduced as guidance for the information selection. From the different modalities, one modality that contributes most to the results is selected as the master modality, which supervises the information selection of the other assistant modalities. The effectiveness of the proposed method is confirmed through breast mass segmentation in MR images of two modalities and better segmentation results are achieved compared to the state-of-the-art methods.

Motivation & Objective

• To address the challenge of noisy and suboptimal feature fusion in multi-modal MRI segmentation using deep learning.
• To improve segmentation accuracy by selectively fusing complementary information from different MRI sequences (e.g., T1-enhanced and T2-weighted).
• To reduce false positives and negatives in breast mass segmentation, especially for spiculated or poorly defined lesions.
• To develop a trainable, attention-based fusion mechanism that mimics knowledge distillation by using one modality to supervise feature selection in others.
• To validate the method on a real clinical dataset of 313 patients with two MRI sequences, demonstrating superiority over existing fusion strategies.

Proposed method

• A multi-stream U-Net architecture is used, with one modality designated as the master stream and others as assistant streams.
• An attention block is applied to the master modality to generate supervision signals that guide feature selection in both master and assistant streams.
• The attention maps from the master modality are used to weight and refine features from all modalities at multiple encoder layers.
• Feature fusion is performed via learnable attention-based concatenation or element-wise weighting, replacing simple summation or concatenation.
• The network is trained end-to-end with a supervised loss, where the master modality's predictions guide the training of assistant modality streams.
• The method is extendable to other multi-modal medical imaging tasks by redefining the master modality based on clinical relevance or performance.

Experimental results

Research questions

• RQ1 Can attention-based, master-modality-guided feature fusion improve segmentation accuracy in multi-modal MRI compared to standard concatenation or summation strategies?
• RQ2 Does using a single high-performing modality (e.g., T1-enhanced) as a supervisory signal enhance feature learning in lower-performing but complementary modalities (e.g., T2-weighted)?
• RQ3 Can the proposed method reduce false positives and false negatives in breast mass segmentation, particularly for challenging, spiculated masses?
• RQ4 How does the attention mechanism localize relevant regions across different modalities, and does it suppress irrelevant or noisy features?
• RQ5 Is the proposed method generalizable to other multi-modal medical image segmentation tasks beyond breast MRI?

Key findings

• The proposed method achieved a Dice similarity coefficient of 84.4±0.7, outperforming all baseline methods including FuseNet, FuseUNet, and late/early fusion baselines.
• The method reduced false positives by 10–15% compared to baseline models, as visually confirmed in example cases (e.g., second and third examples in Fig. 4).
• The attention maps from the T1-enhanced modality successfully localized mass regions (red arrows in Fig. 5), while the T2-weighted stream often highlighted irrelevant areas (blue arrows), validating the need for supervision.
• The proposed method achieved the lowest false negative rate (30.9±1.6%) among all methods, indicating improved detection of small or irregular masses.
• The use of the attention block in the proposed method led to a 1.3% increase in Dice score compared to FuseUNet + SA, demonstrating the effectiveness of the supervision mechanism.
• The model maintained a compact parameter count (26.7M), showing that performance gains were achieved without significant computational overhead.