[Paper Review] MMFNet: A Multi-modality MRI Fusion Network for Segmentation of Nasopharyngeal Carcinoma

Huai Chen, Yuxiao Qi+6 more|arXiv (Cornell University)|Dec 25, 2018

Brain Tumor Detection and Classification68 references4 citations

TL;DR

This paper proposes MMFNet, a 3D multi-encoder fusion network that integrates T1, T2, and contrast-enhanced T1 MRI modalities to improve automatic segmentation of nasopharyngeal carcinoma (NPC). By employing a 3D-CBAM for feature recalibration and a residual fusion block for balanced feature fusion, along with a self-transfer training strategy, MMFNet achieves high-accuracy, time-efficient NPC segmentation, reducing processing time from 10–20 minutes (manual) to ~9 seconds per patient.

ABSTRACT

Segmentation of nasopharyngeal carcinoma (NPC) from Magnetic Resonance Images (MRI) is a crucial prerequisite for NPC radiotherapy. However, manually segmenting of NPC is time-consuming and labor-intensive. Additionally, single-modality MRI generally cannot provide enough information for its accurate delineation. Therefore, a multi-modality MRI fusion network (MMFNet) based on three modalities of MRI (T1, T2 and contrast-enhanced T1) is proposed to complete accurate segmentation of NPC. The backbone of MMFNet is designed as a multi-encoder-based network, consisting of several encoders to capture modality-specific features and one single decoder to fuse them and obtain high-level features for NPC segmentation. A fusion block is presented to effectively fuse features from multi-modality MRI. It firstly recalibrates low-level features captured from modality-specific encoders to highlight both informative features and regions of interest, then fuses weighted features by a residual fusion block to keep balance between fused ones and high-level features from decoder. Moreover, a training strategy named self-transfer, which utilizes pre-trained modality-specific encoders to initialize multi-encoder-based network, is proposed to make full mining of information from different modalities of MRI. The proposed method based on multi-modality MRI can effectively segment NPC and its advantages are validated by extensive experiments.

Motivation & Objective

• To address the limitations of manual NPC segmentation, which is time-consuming and dependent on clinician expertise.
• To overcome the insufficiency of single-modality MRI in capturing comprehensive tissue characteristics for accurate NPC delineation.
• To develop an end-to-end deep learning framework that effectively fuses multi-modality MRI for improved 3D segmentation of NPC.
• To introduce a novel training strategy—self-transfer—that leverages pre-trained modality-specific encoders to enhance feature learning in a multi-encoder architecture.
• To validate the superiority of multi-modality fusion over single- or dual-modality approaches in NPC segmentation performance.

Proposed method

• MMFNet employs a multi-encoder backbone, with separate encoders for T1, T2, and contrast-enhanced T1 MRI to extract modality-specific features.
• A fusion block integrates features from multiple modalities using a 3D-CBAM (Convolutional Block Attention Module) to recalibrate channel and spatial attention, emphasizing informative features and regions of interest.
• The residual fusion block (RFBlock) combines re-weighted features from the 3D-CBAM with high-level decoder features to maintain feature balance and preserve spatial context.
• A self-transfer training strategy initializes the multi-encoder network by transferring knowledge from three pre-trained, modality-specific encoder-decoder networks.
• The network uses 3D convolutional layers throughout, enabling full 3D feature learning and preserving spatial relationships across slices.
• The decoder path reconstructs the segmentation mask using skip connections from the multi-encoder features, enabling precise boundary localization.

Experimental results

Research questions

• RQ1 Can fusing T1, T2, and contrast-enhanced T1 MRI modalities significantly improve the accuracy of NPC segmentation compared to single-modality approaches?
• RQ2 How effective is the proposed 3D-CBAM and residual fusion block in enhancing feature representation and reducing noise in multi-modality fusion?
• RQ3 Does the self-transfer training strategy improve convergence and segmentation performance in a multi-encoder network for NPC?
• RQ4 To what extent does the use of 3D features outperform 2D slice-based methods in capturing anatomical context for NPC delineation?
• RQ5 What is the optimal combination of MRI modalities for NPC segmentation, and does including all three (T1, T2, CET1) yield the best results?

Key findings

• MMFNet achieves superior segmentation performance compared to U-Net, 3D U-Net, and other multi-modality methods, demonstrating state-of-the-art accuracy in NPC segmentation.
• The inclusion of all three MRI modalities—T1, T2, and contrast-enhanced T1—produces the best segmentation results, confirming the value of multi-modality fusion.
• The 3D-CBAM with multiple MLPs and std-pooling integration yields the highest performance, indicating its effectiveness in capturing diverse global feature distributions.
• The self-transfer training strategy significantly improves segmentation metrics across all evaluated configurations, confirming its value in initializing multi-encoder networks.
• MMFNet reduces NPC segmentation time to approximately 9 seconds per patient, a dramatic improvement over manual segmentation (10–20 minutes).
• Visualizations show that attention maps dynamically focus on tumor regions during training, reducing false positives and improving localization accuracy.