[Paper Review] Cross-Modal Information Maximization for Medical Imaging: CMIM

Tristan Sylvain, Francis Dutil+5 more|arXiv (Cornell University)|Oct 20, 2020

Radiomics and Machine Learning in Medical Imaging28 references4 citations

TL;DR

This paper proposes CMIM, a cross-modal information maximization framework that enhances medical image representation learning by maximizing mutual information between multiple imaging modalities (e.g., X-ray, MRI, reports) during training. By doing so, it improves model robustness to modality dropping at inference, outperforming state-of-the-art baselines in both chest X-ray classification and brain tumor segmentation, especially when weaker modalities are used.

ABSTRACT

In hospitals, data are siloed to specific information systems that make the same information available under different modalities such as the different medical imaging exams the patient undergoes (CT scans, MRI, PET, Ultrasound, etc.) and their associated radiology reports. This offers unique opportunities to obtain and use at train-time those multiple views of the same information that might not always be available at test-time. In this paper, we propose an innovative framework that makes the most of available data by learning good representations of a multi-modal input that are resilient to modality dropping at test-time, using recent advances in mutual information maximization. By maximizing cross-modal information at train time, we are able to outperform several state-of-the-art baselines in two different settings, medical image classification, and segmentation. In particular, our method is shown to have a strong impact on the inference-time performance of weaker modalities.

Motivation & Objective

• To address the challenge of modality dropping in real-world medical imaging, where some imaging modalities may be missing at test time.
• To improve representation learning in multi-modal medical data by leveraging mutual information maximization across different imaging modalities.
• To enhance model robustness and performance, particularly for weaker or less informative modalities, by ensuring discriminative features are preserved across all modalities during training.
• To develop a generalizable framework that does not require architectural constraints or modality-specific design, enabling broader applicability across medical imaging tasks.

Proposed method

• The method trains a deep neural network to maximize mutual information between local and global representations of different modalities (e.g., CT, MRI, reports) using a contrastive learning objective.
• It employs a dual-branch encoder structure: one for per-modality local features and one for per-modality global features, with a shared multi-modal global embedding.
• The framework uses mutual information estimation via a neural estimator (inspired by Deep InfoMax and AM-DIM) to optimize local-local and local-global contrastive losses.
• For classification, it includes a local-global mutual information loss; for segmentation, it omits local features to prioritize spatial context.
• The model is trained end-to-end with a task-specific loss (e.g., cross-entropy for classification or segmentation) and mutual information losses.
• It is designed to be modular and general, allowing any combination of modalities and not requiring shared weights or fixed architectural constraints.

Experimental results

Research questions

• RQ1 Can maximizing cross-modal mutual information during training improve model generalization and robustness when certain modalities are missing at inference time?
• RQ2 How does mutual information maximization between multi-modal representations affect performance in medical image classification and segmentation tasks?
• RQ3 Does the proposed method enhance the utility of weaker or less informative modalities (e.g., low-signal MRI sequences) during inference?
• RQ4 Can a mutual information-based framework outperform existing state-of-the-art multi-modal learning methods without architectural constraints like shared weights?
• RQ5 To what extent does CMIM improve performance when only a single modality is available at test time, especially when that modality is suboptimal?

Key findings

• CMIM outperformed both TieNet and ResNet50 on the Open-I chest X-ray classification task, achieving higher AUC, particularly in scenarios with limited training data.
• The method significantly mitigated the issue of modality imbalance, where models tend to ignore image data when reports are sparse, by enforcing mutual information between modalities.
• On the BRATS-2015 segmentation task, CMIM achieved state-of-the-art performance, surpassing Hemis, Mean, and MLP baselines across all metrics.
• The model showed the strongest gains on weaker modalities—such as FLAIR and T1W MRI sequences—demonstrating enhanced robustness when discriminative signal is low.
• The ablation study confirmed that local-global mutual information loss improved performance in classification, while it was less effective in segmentation, where global features were more critical.
• The results validate that CMIM effectively preserves discriminative information across modalities, enabling strong inference even when only one modality is available at test time.