[Paper Review] MRI Pulse Sequence Integration for Deep-Learning Based Brain Metastasis Segmentation

Darvin Yi, Endre Grøvik+10 more|arXiv (Cornell University)|Dec 18, 2019

Advanced MRI Techniques and Applications46 references4 citations

TL;DR

This paper proposes a deep learning framework for brain metastasis segmentation that integrates multiple MRI pulse sequences using input-level dropout to enable robust inference on missing sequences. The method achieves state-of-the-art performance on small datasets and generalizes effectively across institutions without retraining, while visualization reveals consistent, balanced feature learning across sequences.

ABSTRACT

Magnetic resonance (MR) imaging is an essential diagnostic tool in clinical medicine. Recently, a variety of deep learning methods have been applied to segmentation tasks in medical images, with promising results for computer-aided diagnosis. For MR images, effectively integrating different pulse sequences is important to optimize performance. However, the best way to integrate different pulse sequences remains unclear. In this study, we evaluate multiple architectural features and characterize their effects in the task of metastasis segmentation. Specifically, we consider (1) different pulse sequence integration schemas, (2) different modes of weight sharing for parallel network branches, and (3) a new approach for enabling robustness to missing pulse sequences. We find that levels of integration and modes of weight sharing that favor low variance work best in our regime of small data (n = 100). By adding an input-level dropout layer, we could preserve the overall performance of these networks while allowing for inference on inputs with missing pulse sequence. We illustrate not only the generalizability of the network but also the utility of this robustness when applying the trained model to data from a different center, which does not use the same pulse sequences. Finally, we apply network visualization methods to better understand which input features are most important for network performance. Together, these results provide a framework for building networks with enhanced robustness to missing data while maintaining comparable performance in medical imaging applications.

Motivation & Objective

• To identify optimal architectures and integration strategies for combining multiple MRI pulse sequences in deep learning-based brain metastasis segmentation.
• To address the challenge of model sensitivity to missing pulse sequences during inference, especially when deploying models across institutions with differing protocols.
• To develop a single, unified network that maintains high performance across diverse input configurations, including partial or missing sequences.
• To analyze and visualize how different pulse sequences contribute to model decisions using gradient-based saliency maps.
• To evaluate the consistency and stability of training dynamics when using input-level dropout for sequence integration.

Proposed method

• Employed a 3D U-Net-based segmentation network with parallel branches for each MRI pulse sequence (T1 post-contrast, T1 pre-contrast, FLAIR).
• Implemented input-level dropout by randomly masking entire pulse sequence inputs during training, simulating missing data at inference.
• Explored multiple integration schemas, including early fusion (concatenation at input), late fusion (early feature concatenation), and feature subtraction between pre- and post-contrast sequences.
• Used weight sharing across parallel branches to reduce parameters and encourage balanced feature learning, with ablation on sharing modes.
• Applied gradient-based saliency visualization (Grad-CAM) to analyze which input features and sequences were most influential during training and inference.
• Trained models on Stanford data (n=100) and evaluated generalization on independent Oslo University Hospital data with different pulse sequence protocols.

Experimental results

Research questions

• RQ1 Which pulse sequence integration schema (e.g., early, late, or subtraction-based fusion) yields the best segmentation performance in a small-data regime (n=100)?
• RQ2 How does input-level dropout during training affect model robustness when tested on inputs missing one or more pulse sequences?
• RQ3 Can a single model trained on one institution’s data generalize effectively to another institution’s data with different pulse sequence availability?
• RQ4 How does input-level dropout influence training consistency and gradient behavior across different pulse sequences?
• RQ5 Which MRI pulse sequences are most predictive of metastasis segmentation decisions, as revealed by saliency analysis?

Key findings

• The subtraction-based integration of pre- and post-contrast T1-weighted features yielded the best segmentation performance, suggesting that modeling the contrast enhancement difference improves detection.
• Models with lower variance in performance across folds outperformed those with higher variance, indicating that stability is a key factor in small-data regimes.
• Input-level dropout enabled a single model to maintain high performance (DSC ≈ 0.78) on full input sets and rival models trained specifically on each subset of sequences, even when sequences were missing.
• The dropout-trained model generalized effectively to Oslo University Hospital data—where pulse sequences differed from Stanford—achieving comparable performance to institution-specific models.
• Saliency analysis revealed that FLAIR sequences contributed least to segmentation decisions, likely due to expert annotations not including edema, and that dropout training led to more balanced and consistent feature learning across sequences.
• Dropout training produced a more stable and predictable gradient behavior over time, with consistent saliency patterns across training runs, unlike non-dropout models which showed high variability in feature importance.