[Paper Review] Multimodal Representation Learning via Maximization of Local Mutual Information

Ruizhi Liao, Daniel Moyer+6 more|arXiv (Cornell University)|Mar 7, 2021

Multimodal Machine Learning ApplicationsComputer Science39 references32 citations

TL;DR

This paper proposes a multimodal representation learning framework that maximizes local mutual information (MI) between image patches and sentence-level text embeddings to improve downstream image classification. By training image and text encoders to align local visual features with corresponding textual descriptions—using neural network discriminators for MI estimation—it achieves state-of-the-art performance on medical imaging benchmarks, outperforming global MI and supervised baselines, especially when fine-tuned on downstream tasks.

ABSTRACT

We propose and demonstrate a representation learning approach by maximizing the mutual information between local features of images and text. The goal of this approach is to learn useful image representations by taking advantage of the rich information contained in the free text that describes the findings in the image. Our method trains image and text encoders by encouraging the resulting representations to exhibit high local mutual information. We make use of recent advances in mutual information estimation with neural network discriminators. We argue that the sum of local mutual information is typically a lower bound on the global mutual information. Our experimental results in the downstream image classification tasks demonstrate the advantages of using local features for image-text representation learning.

Motivation & Objective

• To improve image representation learning in medical imaging by leveraging free-text radiology reports as supervisory signals.
• To address the limitation of global mutual information in capturing fine-grained image-text alignments by focusing on local features.
• To develop a method that better aligns image regions with their corresponding textual descriptions for improved downstream classification.
• To demonstrate that local MI maximization yields more flexible and task-adaptive representations than global MI or supervised pretraining alone.

Proposed method

• Proposes maximizing mutual information (MI) between local image features (e.g., 4×4×512 patches) and sentence-level text features extracted from radiology reports.
• Uses neural network discriminators to estimate and optimize lower bounds on MI, employing MINE and CPC estimators for robustness.
• Trains image and text encoders jointly with the MI discriminator by optimizing a contrastive objective over matched and shuffled image-text pairs.
• Applies local MI maximization by selecting the image patch with the highest MI to a given sentence and optimizing that alignment.
• Employs a 5-block ResNet for local MI and a 6-block ResNet for global MI; uses clinical BERT for text encoding and multilayer perceptrons for MI discrimination.
• Fine-tunes the image encoder on downstream classification tasks, showing improved performance over frozen or globally aligned features.

Experimental results

Research questions

• RQ1 Does maximizing local mutual information between image patches and sentence-level text descriptions lead to better image representations than global MI or supervised learning?
• RQ2 How does local MI maximization compare to global MI in terms of downstream classification performance on medical imaging tasks?
• RQ3 Can local MI learning improve performance even when the image encoder is frozen, or is fine-tuning essential for its benefits?
• RQ4 Is the performance gain from local MI robust to the choice of MI estimator (e.g., MINE vs. CPC)?

Key findings

• The local MI approach with fine-tuned encoders achieves an average AUC of 0.88 on the EdemaSeverity ordinal classification task, significantly outperforming the next best method (0.85 for global MI).
• On the Pathology9 binary classification benchmark, the tuned local MI method achieves an average AUC of 0.84, compared to 0.81 for global MI, demonstrating consistent gains across multiple pathologies.
• Even with frozen encoders, the local MI method achieves AUC scores comparable to fully supervised learning on several tasks, indicating strong unsupervised feature learning.
• The performance gain from local MI is most pronounced when the image encoder is fine-tuned, suggesting that local MI produces more adaptable and task-specific representations.
• The method shows minimal sensitivity to the choice of MI estimator, with both MINE and CPC yielding similar performance across tasks.
• Local MI maximization leads to better optimization landscape due to lower-dimensional representations and more training samples per image, enhancing learning efficiency and generalization.