[Paper Review] ModDrop++: A Dynamic Filter Network with Intra-subject Co-training for Multiple Sclerosis Lesion Segmentation with Missing Modalities

Han Liu, Yubo Fan+7 more|arXiv (Cornell University)|Mar 7, 2022

Microbial infections and disease research4 citations

TL;DR

ModDrop++ proposes a dynamic filter network with intra-subject co-training to improve multiple sclerosis lesion segmentation under missing MRI modalities. By introducing a plug-and-play dynamic head for adaptive feature learning and co-training between full- and missing-modality data from the same subject, it achieves state-of-the-art performance across all missing modality conditions without architectural changes.

ABSTRACT

Multiple Sclerosis (MS) is a chronic neuroinflammatory disease and multi-modality MRIs are routinely used to monitor MS lesions. Many automatic MS lesion segmentation models have been developed and have reached human-level performance. However, most established methods assume the MRI modalities used during training are also available during testing, which is not guaranteed in clinical practice. Previously, a training strategy termed Modality Dropout (ModDrop) has been applied to MS lesion segmentation to achieve the state-of-the-art performance with missing modality. In this paper, we present a novel method dubbed ModDrop++ to train a unified network adaptive to an arbitrary number of input MRI sequences. ModDrop++ upgrades the main idea of ModDrop in two key ways. First, we devise a plug-and-play dynamic head and adopt a filter scaling strategy to improve the expressiveness of the network. Second, we design a co-training strategy to leverage the intra-subject relation between full modality and missing modality. Specifically, the intra-subject co-training strategy aims to guide the dynamic head to generate similar feature representations between the full- and missing-modality data from the same subject. We use two public MS datasets to show the superiority of ModDrop++. Source code and trained models are available at https://github.com/han-liu/ModDropPlusPlus.

Motivation & Objective

• Address the clinical challenge of inconsistent MRI modality availability in multiple sclerosis lesion segmentation.
• Overcome limitations of Modality Dropout (ModDrop), such as limited network expressiveness and lack of intra-subject knowledge transfer.
• Develop a plug-and-play method that improves generalization across arbitrary missing modality combinations without modifying existing network architectures.
• Minimize the performance gap between unified models and modality-specific models for robust clinical deployment.
• Leverage inter-subject relationships to enhance feature representation consistency between full and missing modality inputs from the same patient.

Proposed method

• Introduce a plug-and-play dynamic head that generates filter scaling matrices conditioned on the missing modality pattern, enabling adaptive feature learning.
• Apply a filter scaling strategy to dynamically adjust the first convolutional layer’s parameters based on input modality availability.
• Design an intra-subject co-training strategy to align feature representations between full-modality and missing-modality inputs from the same subject.
• Train the dynamic head using contrastive loss to encourage similarity in feature maps from the same subject despite modality gaps.
• Integrate the dynamic head and co-training into a unified network trained under the ModDrop paradigm, preserving model architecture flexibility.
• Use two public MS datasets (ISBI 2015 and MSSEG) to validate performance across diverse missing modality scenarios.

Experimental results

Research questions

• RQ1 Can a dynamic filter mechanism improve model expressiveness in MS lesion segmentation under missing modality conditions?
• RQ2 Does intra-subject co-training between full- and missing-modality data enhance feature representation consistency and segmentation accuracy?
• RQ3 Can ModDrop++ achieve performance comparable to modality-specific models while maintaining a single unified architecture?
• RQ4 How does the combination of dynamic filtering and co-training outperform standard ModDrop across various missing modality configurations?
• RQ5 To what extent does the method generalize across datasets with varying modality availability and subject counts?

Key findings

• ModDrop++ achieved the highest Dice Similarity Coefficient (DSC) of 0.793 on the ISBI 2015 dataset when all modalities were available, outperforming baseline ModDrop and modality-specific models.
• With only T1w available, ModDrop++ achieved a DSC of 0.710, significantly outperforming ModDrop (0.684) and reducing false positives and negatives.
• When FLAIR was missing, ModDrop++ maintained a DSC of 0.772, demonstrating robustness to the absence of this critical modality.
• The combination of the dynamic head and intra-subject co-training (ModDrop++) reduced the performance gap between unified and modality-specific models, especially in low-modality regimes.
• On the MSSEG dataset, ModDrop++ achieved a DSC of 0.776 with all modalities present and maintained strong performance (0.748) with only T1w, showing consistent generalization.
• Qualitative results showed fewer false positives and false negatives in ModDrop++ compared to ModDrop, particularly in single-modality scenarios.