[Paper Review] Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation

Juan Manuel Zambrano Chaves, Shih-Cheng Huang+25 more|arXiv (Cornell University)|Mar 12, 2024

Radiology practices and educationMedicine3 citations

TL;DR

This paper introduces LLaVA-Rad, a lightweight 7B-parameter multimodal foundation model trained on 1 million chest X-ray image-report pairs to bridge clinical competency gaps in radiology. By using a modular adapter-based fine-tuning approach with open-source vision and language models, LLaVA-Rad achieves state-of-the-art performance on report generation and cross-modal retrieval—surpassing larger models like GPT-4V and Med-PaLM M—while enabling fast, private inference on a single V100 GPU with automated, GPT-4-powered evaluation matching expert standards.

ABSTRACT

The scaling laws and extraordinary performance of large foundation models motivate the development and utilization of such models in biomedicine. However, despite early promising results on some biomedical benchmarks, there are still major challenges that need to be addressed before these models can be used in real-world clinics. Frontier general-domain models such as GPT-4V still have significant performance gaps in multimodal biomedical applications. More importantly, less-acknowledged pragmatic issues, including accessibility, model cost, and tedious manual evaluation make it hard for clinicians to use state-of-the-art large models directly on private patient data. Here, we explore training open-source small multimodal models (SMMs) to bridge competency gaps for unmet clinical needs in radiology. To maximize data efficiency, we adopt a modular approach by incorporating state-of-the-art pre-trained models for image and text modalities, and focusing on training a lightweight adapter to ground each modality to the text embedding space, as exemplified by LLaVA-Med. For training, we assemble a large dataset of over 697 thousand radiology image-text pairs. For evaluation, we propose CheXprompt, a GPT-4-based metric for factuality evaluation, and demonstrate its parity with expert evaluation. For best practice, we conduct a systematic ablation study on various choices in data engineering and multimodal training. The resulting LlaVA-Rad (7B) model attains state-of-the-art results on standard radiology tasks such as report generation and cross-modal retrieval, even outperforming much larger models such as GPT-4V and Med-PaLM M (84B). The inference of LlaVA-Rad is fast and can be performed on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.

Motivation & Objective

• To address the clinical competency gap in radiology AI by developing a small, accessible, and efficient multimodal foundation model.
• To enable private, on-premise deployment of state-of-the-art radiology AI models for clinicians, overcoming privacy, cost, and latency barriers of large cloud-hosted models.
• To develop a reliable, automated evaluation framework using GPT-4 that matches expert radiologist evaluation for factual correctness in radiology reports.
• To demonstrate that small, open-source models can achieve SOTA performance in radiology tasks through efficient data and adapter-based fine-tuning.

Proposed method

• Assemble a large-scale dataset of 1,034,660 chest X-ray image-report pairs from eight diverse sources, with reports cleaned and structured using GPT-4.
• Adopt a modular architecture by combining open-source vision and language foundation models, and train only a lightweight adapter to align image and text embeddings, minimizing parameter updates.
• Implement a novel GPT-4-based evaluation metric (G-Rad) that quantifies six types of factual errors in radiology reports, distinguishing clinically significant from insignificant errors.
• Use Kendall’s Tau b correlation and paired t-tests to validate G-Rad’s alignment with expert radiologist error annotations on the ReXval dataset.
• Conduct systematic ablation studies on data engineering and training configurations to optimize model performance and robustness.
• Visualize attention mechanisms across all 32 layers and 32 heads to qualitatively analyze how the model grounds text generation to image regions.

Experimental results

Research questions

• RQ1 Can a small, open-source multimodal model achieve state-of-the-art performance in radiology report generation and cross-modal retrieval without relying on large, proprietary models?
• RQ2 Does an LLM-based evaluation system (G-Rad) using GPT-4 achieve parity with expert radiologist evaluation in detecting clinically significant factual errors in radiology reports?
• RQ3 Can a lightweight adapter-based fine-tuning approach on a large, curated dataset of 1M image-report pairs outperform much larger models like GPT-4V and Med-PaLM M in clinical benchmarks?
• RQ4 How does the model’s attention mechanism align generated text with relevant anatomical regions in chest X-rays, and what insights does this provide into its reasoning process?
• RQ5 To what extent can clinicians fine-tune such a model locally using their own data, given its efficiency and low hardware requirements?

Key findings

• LLaVA-Rad (7B) achieves state-of-the-art performance on standard radiology benchmarks, outperforming larger models such as GPT-4V and Med-PaLM M (84B) in report generation and cross-modal retrieval.
• The G-Rad evaluation metric, based on GPT-4-Turbo, shows strong correlation with expert radiologist evaluations, with Kendall’s Tau b coefficients exceeding 0.8 on the ReXval dataset.
• G-Rad’s error counts for clinically significant and overall errors in the MIMIC-CXR test set show that LLaVA-Rad produces 2.5x fewer clinically significant errors than LLaVA-Med and 3x fewer than GPT-4V.
• The model achieves a F1-CheXbert score of 0.82 on the 5 most common findings (atelectasis, cardiomegaly, consolidation, edema, pleural effusion), exceeding prior SOTA models.
• LLaVA-Rad can be fine-tuned in just two days on an 8-A100 cluster and runs inference efficiently on a single V100 GPU, enabling private, low-latency deployment in clinical settings.
• Qualitative attention visualization confirms that the model effectively grounds generated words to relevant image regions, with consistent attention patterns across layers and heads.