[Paper Review] Cross-modality Guidance-aided Multi-modal Learning with Dual Attention for MRI Brain Tumor Grading

Dunyuan Xu, Xi Wang+2 more|arXiv (Cornell University)|Jan 17, 2024

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes a cross-modality guidance-aided multi-modal learning framework with dual attention for MRI-based brain tumor grading, using a primary modality to guide feature learning in secondary modalities and applying spatial and slice-wise attention to enhance informative feature extraction. The method achieves an AUC of 0.985 ± 0.019 on BraTS2018 and 0.966 ± 0.021 on BraTS2019, outperforming state-of-the-art methods by a significant margin.

ABSTRACT

Brain tumor represents one of the most fatal cancers around the world, and is very common in children and the elderly. Accurate identification of the type and grade of tumor in the early stages plays an important role in choosing a precise treatment plan. The Magnetic Resonance Imaging (MRI) protocols of different sequences provide clinicians with important contradictory information to identify tumor regions. However, manual assessment is time-consuming and error-prone due to big amount of data and the diversity of brain tumor types. Hence, there is an unmet need for MRI automated brain tumor diagnosis. We observe that the predictive capability of uni-modality models is limited and their performance varies widely across modalities, and the commonly used modality fusion methods would introduce potential noise, which results in significant performance degradation. To overcome these challenges, we propose a novel cross-modality guidance-aided multi-modal learning with dual attention for addressing the task of MRI brain tumor grading. To balance the tradeoff between model efficiency and efficacy, we employ ResNet Mix Convolution as the backbone network for feature extraction. Besides, dual attention is applied to capture the semantic interdependencies in spatial and slice dimensions respectively. To facilitate information interaction among modalities, we design a cross-modality guidance-aided module where the primary modality guides the other secondary modalities during the process of training, which can effectively leverage the complementary information of different MRI modalities and meanwhile alleviate the impact of the possible noise.

Motivation & Objective

• To address the limitations of uni-modal models and noisy fusion in multi-modal MRI brain tumor grading.
• To reduce performance degradation caused by noise in traditional modality fusion strategies.
• To improve model robustness and diagnostic accuracy by leveraging complementary information across MRI sequences.
• To develop a guidance mechanism that selectively enhances informative features from secondary modalities using the primary modality as a reference.
• To achieve high performance with minimal pre-processing, relying only on raw MRI data.

Proposed method

• Uses ResNet Mix Convolution as a lightweight backbone for efficient feature extraction across MRI modalities.
• Introduces dual attention mechanisms to capture semantic interdependencies in both spatial and slice dimensions.
• Designs a cross-modality guidance module where the highest-performing modality guides learning in lower-performing modalities.
• Applies high-level features from the primary modality to guide low-level features in secondary modalities, bridging semantic gaps.
• Employs an accumulative learning strategy to preserve previously learned valuable features across training stages.
• Utilizes a two-stage training process to refine feature learning with guidance and attention mechanisms.

Experimental results

Research questions

• RQ1 Can a primary MRI modality effectively guide the learning process in secondary modalities to reduce noise and improve performance?
• RQ2 How does dual attention in spatial and slice dimensions enhance feature representation for tumor grading?
• RQ3 Does cross-modality guidance mitigate performance degradation caused by naive fusion of multi-modal MRI data?
• RQ4 Can the proposed method achieve state-of-the-art performance without relying on pre-processing like tumor ROI annotation or radiomics extraction?
• RQ5 How robust is the model across different datasets with varying data distributions?

Key findings

• The proposed method achieves an AUC of 0.985 ± 0.019 on the BraTS2018 dataset, significantly outperforming both uni-modal and multi-modal baselines.
• On the BraTS2019 dataset, the method achieves an AUC of 0.966 ± 0.021, demonstrating strong generalization and robustness.
• Ablation studies confirm that both the cross-modality guidance and dual attention components are essential for optimal performance.
• The model outperforms existing state-of-the-art methods without requiring tumor ROI ground truth or radiomics feature extraction.
• The framework shows improved robustness through three-fold cross-validation, indicating consistent performance across data splits.
• Despite strong performance, the model's large parameter count and two-stage training process limit efficiency and re-implementation ease.