[Paper Review] Advancing human-centric AI for robust X-ray analysis through holistic self-supervised learning

Théo Moutakanni, Piotr Bojanowski+8 more|arXiv (Cornell University)|May 2, 2024

Radiomics and Machine Learning in Medical ImagingMedicine3 citations

TL;DR

RayDINO is a 307M-parameter vision transformer trained via self-supervised DINOv2 on 873k chest X-rays from diverse global datasets. It achieves state-of-the-art performance across 21 benchmarks in classification, segmentation, report generation, and generalization, while demonstrating improved robustness to demographic biases and holistic patient-centric analysis without task-specific fine-tuning of the backbone.

ABSTRACT

AI Foundation models are gaining traction in various applications, including medical fields like radiology. However, medical foundation models are often tested on limited tasks, leaving their generalisability and biases unexplored. We present RayDINO, a large visual encoder trained by self-supervision on 873k chest X-rays. We compare RayDINO to previous state-of-the-art models across nine radiology tasks, from classification and dense segmentation to text generation, and provide an in depth analysis of population, age and sex biases of our model. Our findings suggest that self-supervision allows patient-centric AI proving useful in clinical workflows and interpreting X-rays holistically. With RayDINO and small task-specific adapters, we reach state-of-the-art results and improve generalization to unseen populations while mitigating bias, illustrating the true promise of foundation models: versatility and robustness.

Motivation & Objective

• To develop a foundation model for chest X-ray analysis that enables holistic, patient-centric interpretation beyond narrow task-specific supervision.
• To address the limitations of current AI models in radiology, including poor generalization to unseen populations and biases related to age, sex, and ethnicity.
• To evaluate the robustness and fairness of self-supervised vision foundation models in real-world clinical settings across diverse populations and imaging protocols.
• To demonstrate that a single frozen vision encoder with lightweight task-specific adapters can outperform specialized models across multiple radiology tasks.
• To validate the clinical utility and interpretability of the model through expert radiologist evaluation and comprehensive benchmarking.

Proposed method

• Training a vision transformer (ViT) encoder using the DINOv2 self-supervised learning objective on 873,000 chest X-rays from four public datasets across the USA and Europe.
• Leveraging contrastive self-supervision to learn rich, generalizable representations without requiring any human-annotated labels or paired text.
• Applying the frozen RayDINO encoder directly to nine downstream tasks—including classification, segmentation, and report generation—using only small, trainable adapters.
• Evaluating the model on eleven external datasets from seven countries across four continents to assess zero-shot generalization and demographic robustness.
• Using standardized metrics such as AUROC, mDice, Pearson correlation, and AUPRC across diverse benchmarks, including rare diseases and underrepresented populations.
• Conducting bias analysis across age, sex, and population demographics to assess fairness and model generalization.

Experimental results

Research questions

• RQ1 Can a self-supervised vision foundation model trained on a large, diverse X-ray dataset achieve state-of-the-art performance across multiple radiology tasks without task-specific fine-tuning of the backbone?
• RQ2 How does RayDINO perform in zero-shot generalization across external datasets from different countries and imaging protocols?
• RQ3 To what extent does self-supervised pretraining reduce demographic biases (e.g., by age, sex, or ethnicity) compared to supervised baselines?
• RQ4 Can a single frozen encoder with lightweight adapters provide holistic, patient-centric representations that support diverse clinical tasks?
• RQ5 How does RayDINO compare in interpretability and clinical relevance when evaluated by expert radiologists?

Key findings

• RayDINO achieves state-of-the-art performance on 21 benchmarks across nine radiology tasks, including classification, segmentation, report generation, and generalization to unseen populations.
• The model significantly outperforms previous state-of-the-art models in AUROC (classification), mDice (segmentation), and AUPRC (rare classes), with improvements up to 10% on certain benchmarks.
• RayDINO demonstrates superior generalization to unseen populations and external datasets, achieving higher AUROC on BRAX and PAXRay benchmarks compared to prior models.
• The model shows reduced bias across age and sex subgroups, with more consistent performance on minority demographic groups compared to supervised baselines.
• Expert radiologists validated the interpretability of RayDINO’s representations, confirming clinical relevance and holistic understanding of X-ray findings.
• Despite using a frozen encoder, RayDINO achieves competitive performance with minimal task-specific adaptation, highlighting the versatility and robustness of self-supervised pretraining in medical imaging.