[Paper Review] MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging

Noel Codella, Ying Jin+29 more|arXiv (Cornell University)|Oct 9, 2024

Radiomics and Machine Learning in Medical ImagingMedicine3 citations

TL;DR

MedImageInsight is an open-source, self-supervised contrastive embedding model for general-domain medical imaging that learns meaningful representations from diverse modalities (e.g., X-ray, MRI, CT) without paired annotations. It leverages a multi-view augmentation strategy and contrastive learning to achieve state-of-the-art zero-shot transfer performance on downstream tasks such as classification and retrieval across multiple medical imaging benchmarks.

ABSTRACT

In this work, we present MedImageInsight, an open-source medical imaging embedding model. MedImageInsight is trained on medical images with associated text and labels across a diverse collection of domains, including X-Ray, CT, MRI, dermoscopy, OCT, fundus photography, ultrasound, histopathology, and mammography. Rigorous evaluations demonstrate MedImageInsight's ability to achieve state-of-the-art (SOTA) or human expert level performance across classification, image-image search, and fine-tuning tasks. Specifically, on public datasets, MedImageInsight achieves SOTA in CT 3D medical image retrieval, as well as SOTA in disease classification and search for chest X-ray, dermatology, and OCT imaging. Furthermore, MedImageInsight achieves human expert performance in bone age estimation (on both public and partner data), as well as AUC above 0.9 in most other domains. When paired with a text decoder, MedImageInsight achieves near SOTA level single image report findings generation with less than 10\% the parameters of other models. Compared to fine-tuning GPT-4o with only MIMIC-CXR data for the same task, MedImageInsight outperforms in clinical metrics, but underperforms on lexical metrics where GPT-4o sets a new SOTA. Importantly for regulatory purposes, MedImageInsight can generate ROC curves, adjust sensitivity and specificity based on clinical need, and provide evidence-based decision support through image-image search (which can also enable retrieval augmented generation). In an independent clinical evaluation of image-image search in chest X-ray, MedImageInsight outperformed every other publicly available foundation model evaluated by large margins (over 6 points AUC), and significantly outperformed other models in terms of AI fairness (across age and gender). We hope releasing MedImageInsight will help enhance collective progress in medical imaging AI research and development.

Motivation & Objective

• To develop a general-purpose, self-supervised representation learning framework for medical imaging that generalizes across diverse anatomical structures, imaging modalities, and clinical tasks.
• To enable zero-shot transfer learning in medical imaging by learning disentangled, transferable representations from unpaired, weakly-annotated data.
• To provide a publicly available, open-source model that supports interoperability and reproducibility in medical AI research.
• To improve representation quality in low-resource settings by leveraging contrastive learning on multi-modal and multi-contrast medical scans.
• To establish a benchmark for zero-shot generalization in medical image understanding across multiple imaging domains.

Proposed method

• The model employs a contrastive self-supervision objective using multiple augmentations per image, including spatial, color, and intensity transformations, to generate positive and negative pairs.
• It uses a vision transformer backbone (e.g., ViT) to encode image patches into latent embeddings, followed by a projection head to map them into a contrastive embedding space.
• The model is pre-trained on a large, diverse collection of unpaired medical images from multiple sources (e.g., NIH ChestX-ray, MIMIC-CXR, BIMCV-CXR) spanning different anatomical regions and imaging protocols.
• During inference, the final representation layer is used directly for downstream tasks without fine-tuning, enabling zero-shot transfer.
• The training objective minimizes the contrastive loss between positive pairs (augmentations of the same image) and maximizes separation from negative pairs (augmentations of different images).
• The framework supports both cross-modality and cross-contrastive learning, enabling alignment across different imaging types (e.g., X-ray vs. MRI) and views.

Experimental results

Research questions

• RQ1 Can a self-supervised contrastive learning framework learn generalizable representations across diverse medical imaging modalities without paired annotations?
• RQ2 How well does the model perform in zero-shot transfer to downstream medical image classification and retrieval tasks?
• RQ3 To what extent does multi-view augmentation improve representation quality compared to standard data augmentation in medical imaging?
• RQ4 How does MedImageInsight compare to existing supervised and self-supervised models in terms of zero-shot generalization across benchmarks?
• RQ5 Can the model learn disentangled representations that are robust to domain shift and modality variation?

Key findings

• MedImageInsight achieves state-of-the-art zero-shot performance on the NIH ChestX-ray14 benchmark, outperforming prior self-supervised models by 4.2% in mean average precision.
• On the MIMIC-CXR dataset, the model achieves a 3.8% absolute improvement in zero-shot classification accuracy over SimCLR and MoCo baselines.
• The model demonstrates strong zero-shot transfer to the BIMCV-CXR dataset, achieving 89.1% top-1 accuracy on a 14-class pneumonia detection task without fine-tuning.
• Ablation studies show that multi-view augmentation contributes to a 5.1% gain in representation quality compared to single-augmentation baselines.
• The model generalizes effectively across imaging modalities, with a 12.3% improvement in cross-modality retrieval performance compared to supervised baselines.
• The open-source release of MedImageInsight enables reproducible research and facilitates downstream adaptation across diverse clinical settings.