[Paper Review] ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoders

Shawn Xu, Lin Yang+26 more|arXiv (Cornell University)|Aug 2, 2023

COVID-19 diagnosis using AI18 citations

TL;DR

ELIXR aligns a vision encoder with a frozen LLM to enable zero-shot classification, data-efficient learning, semantic search, VQA, and radiology report QA on chest X-rays using paired images and free-text reports.

ABSTRACT

In this work, we present an approach, which we call Embeddings for Language/Image-aligned X-Rays, or ELIXR, that leverages a language-aligned image encoder combined or grafted onto a fixed LLM, PaLM 2, to perform a broad range of chest X-ray tasks. We train this lightweight adapter architecture using images paired with corresponding free-text radiology reports from the MIMIC-CXR dataset. ELIXR achieved state-of-the-art performance on zero-shot chest X-ray (CXR) classification (mean AUC of 0.850 across 13 findings), data-efficient CXR classification (mean AUCs of 0.893 and 0.898 across five findings (atelectasis, cardiomegaly, consolidation, pleural effusion, and pulmonary edema) for 1% (~2,200 images) and 10% (~22,000 images) training data), and semantic search (0.76 normalized discounted cumulative gain (NDCG) across nineteen queries, including perfect retrieval on twelve of them). Compared to existing data-efficient methods including supervised contrastive learning (SupCon), ELIXR required two orders of magnitude less data to reach similar performance. ELIXR also showed promise on CXR vision-language tasks, demonstrating overall accuracies of 58.7% and 62.5% on visual question answering and report quality assurance tasks, respectively. These results suggest that ELIXR is a robust and versatile approach to CXR AI.

Motivation & Objective

• Leverage routinely collected chest X-ray (CXR) image–report pairs to train a multimodal model that works across diverse radiology tasks.
• Achieve strong zero-shot and data-efficient performance on CXR classification with rich, text-guided outputs.
• Enable multimodal capabilities such as semantic search, visual question answering, and radiology report QA through LLM-alignment.

Proposed method

• Two-stage architecture: ELIXR-C trains a language-aligned image encoder using CLIP-style contrastive learning to align CXR images with radiology reports.
• ELIXR-B grafts a lightweight adapter (Q-Former) between the frozen ELIXR-C encoder and a frozen LLM (PaLM 2-S) to map image embeddings to the LLM’s token space.
• Training is data-efficient by freezing the LLM and vision encoder, training only the adapter.
• Phase 1 uses image-text contrastive learning (ITC), image-grounded text generation (ITG), and image-text matching (ITM) to learn cross-modal representations.
• Phase 2 trains the adapter to generate LLM-aligned tokens and to enable impression generation, then uses the LLM for downstream generation tasks like VQA and report QA.

Experimental results

Research questions

• RQ1 Can ELIXR achieve zero-shot classification performance on chest X-ray findings comparable to state-of-the-art supervised methods?
• RQ2 How data-efficient can CXR classification be when using ELIXR-C and ELIXR-B compared to fully supervised baselines?
• RQ3 Can the ELIXR framework support semantic search and retrieve clinically relevant CXR images with high quality?
• RQ4 Is ELIXR capable of VQA and radiology report quality assurance using an LLM-aligned multimodal pipeline?
• RQ5 Does the approach generalize across multiple datasets and hospital sources?

Key findings

• ELIXR achieves state-of-the-art zero-shot classification with a mean AUC of 0.850 across 13 findings on CheXpert.
• ELIXR-C and ELIXR-B achieve data-efficient linear probe performance, with mean AUC of 0.893 (1% data) and 0.898 (10% data) on five CheXpert findings, surpassing prior methods.
• For semantic search, ELIXR attains 0.76 NDCG@5 across nineteen queries with several perfect retrievals (12 of 19 queries).
• ELIXR-B outperforms ELIXR-C and MedCLIP on retrieval quality, achieving higher precision@5 and NDCG@5 across query groups.
• On VQA and report QA tasks, ELIXR demonstrates overall accuracies of 58.7% (VQA) and 62.5% (report QA) on CXR tasks.
• Compared to existing data-efficient methods, ELIXR requires two orders of magnitude less data to reach similar performance.