[Paper Review] Exploring scalable medical image encoders beyond text supervision

Fernando Pérez‐García, Harshita Sharma+13 more|arXiv (Cornell University)|Jan 19, 2024

Artificial Intelligence in Healthcare and Education8 citations

TL;DR

The paper shows an image-only self-supervised biomedical image encoder (raddino) trained with DINOv2 and masked image modelling can match or surpass text-supervised models across classification, segmentation, and vision–language tasks, while scaling with data and correlating more with clinical information.

ABSTRACT

Language-supervised pre-training has proven to be a valuable method for extracting semantically meaningful features from images, serving as a foundational element in multimodal systems within the computer vision and medical imaging domains. However, the computed features are limited by the information contained in the text, which is particularly problematic in medical imaging, where the findings described by radiologists focus on specific observations. This challenge is compounded by the scarcity of paired imaging-text data due to concerns over leakage of personal health information. In this work, we fundamentally challenge the prevailing reliance on language supervision for learning general-purpose biomedical imaging encoders. We introduce RAD-DINO, a biomedical image encoder pre-trained solely on unimodal biomedical imaging data that obtains similar or greater performance than state-of-the-art biomedical language-supervised models on a diverse range of benchmarks. Specifically, the quality of learned representations is evaluated on standard imaging tasks (classification and semantic segmentation), and a vision-language alignment task (text report generation from images). To further demonstrate the drawback of language supervision, we show that features from RAD-DINO correlate with other medical records (e.g., sex or age) better than language-supervised models, which are generally not mentioned in radiology reports. Finally, we conduct a series of ablations determining the factors in RAD-DINO's performance; notably, we observe that RAD-DINO's downstream performance scales well with the quantity and diversity of training data, demonstrating that image-only supervision is a scalable approach for training a foundational biomedical image encoder. Model weights of RAD-DINO trained on publicly available datasets are available at https://huggingface.co/microsoft/rad-dino.

Motivation & Objective

• Motivate reducing reliance on language supervision for biomedical image encoders due to limited image–text data and PHI concerns.
• Propose raddino, an image-only encoder trained with DINOv2 and masked image modelling (MIM) for global and local feature learning.
• Evaluate raddino on image classification, semantic segmentation, and text report generation to test uni- and multimodal capabilities.
• Demonstrate that image-only representations can correlate more strongly with patient demographics and EHR-like information than language-supervised models.

Proposed method

• Pre-train raddino using DINOv2 with a hybrid objective: masked image modelling (MIM) for patch-level prediction and image-level contrastive learning with multi-crop views.
• Start from DINOv2 ViT-B, continue pre-training on large-scale, diverse radiology image datasets (Multi-CXR) with domain-transfer experiments from general-domain weights.
• Compare against image-text and multimodal baselines (e.g., CLIP variants, BiomedCLIP, BioViL-T, MRM) using linear probing on external CXR datasets.
• Evaluate on image classification (VinDr-CXR, CANDID-PTX, RSNA Pneumonia), semantic segmentation (CANDID-PTX, MIMIC-CXR-derived datasets), and a vision–language task (text report generation on MIMIC-CXR).
• Investigate ablations: effect of input resolution, weight initialization, and training data size/diversity on downstream performance.

Experimental results

Research questions

• RQ1 Can image-only self-supervised learning match or exceed text-supervised biomedical encoders across standard imaging tasks?
• RQ2 Does raddino scale favorably with training data size, diversity, and higher input resolution for both global and local (patch-level) tasks?
• RQ3 Do image-only encoders produce representations more aligned with patient demographics and non-report clinical information than language-supervised counterparts?
• RQ4 What is the impact of MIM and domain-transfer pre-training on segmentation and vision–language generation performance?
• RQ5 Is a purely image-based pre-training approach viable for unified foundational biomedical image encoders without relying on image–text data?

Key findings

• raddino matches or surpasses state-of-the-art language-supervised models on diverse biomedical benchmarks for image classification and segmentation.
• On VinDr-CXR, raddino achieves the highest Agg AUPRC (66.63) and outperforms CLIP and other baselines across findings.
• In CANDID-PTX and RSNA Pneumonia, raddino attains strong results, notably excelling on pneumothorax and chest tubes in PTX-related tasks.
• For vision–language generation, raddino-based encoders yield superior ROUGE-L, BLEU-4, RG ER, and Macro-F1-14 scores, indicating higher factuality and clinical accuracy in generated findings.
• Ablations show performance scales with larger, more diverse training data and higher input resolution; domain-transfer from general-domain models helps but continual in-domain pre-training provides further gains.
• raddino encodings correlate with broader clinical information (e.g., demographics) more than language-supervised models, suggesting broader applicability to multimodal clinical tasks.