[Paper Review] A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretation

Zhihong Chen, Maya Varma+21 more|arXiv (Cornell University)|Jan 22, 2024

COVID-19 diagnosis using AI24 citations

TL;DR

The paper introduces CheXinstruct, CheXagent, and CheXbench to build and evaluate a vision-language foundation model for chest X-ray interpretation, achieving strong gains over baselines and assessing fairness.

ABSTRACT

Over 1.4 billion chest X-rays (CXRs) are performed annually due to their cost-effectiveness as an initial diagnostic test. This scale of radiological studies provides a significant opportunity to streamline CXR interpretation and documentation. While foundation models are a promising solution, the lack of publicly available large-scale datasets and benchmarks inhibits their iterative development and real-world evaluation. To overcome these challenges, we constructed a large-scale dataset (CheXinstruct), which we utilized to train a vision-language foundation model (CheXagent). We systematically demonstrated competitive performance across eight distinct task types on our novel evaluation benchmark (CheXbench). Beyond technical validation, we assessed the real-world utility of CheXagent in directly drafting radiology reports. Our clinical assessment with eight radiologists revealed a 36% time saving for residents using CheXagent-drafted reports, while attending radiologists showed no significant time difference editing resident-drafted or CheXagent-drafted reports. The CheXagent-drafted reports improved the writing efficiency of both radiology residents and attending radiologists in 81% and 61% of cases, respectively, without loss of quality. Overall, we demonstrate that CheXagent can effectively perform a variety of CXR interpretation tasks and holds potential to assist radiologists in routine clinical workflows.

Motivation & Objective

• Create a large-scale instruction-tuning dataset (CheXinstruct) for chest X-rays spanning multiple tasks.
• Develop an 8B-parameter vision-language foundation model (CheXagent) for CXR interpretation.
• Bridge vision and language with a training pipeline including a clinical LLM, CXR vision encoder, and a bridging module.
• Establish CheXbench to evaluate FM performance across image perception and textual understanding in CXRs.
• Assess model fairness across sex, race, and age to improve transparency.

Proposed method

• Assemble CheXinstruct from 34 tasks and 65 datasets into 6.1M instruction-answer triplets for CXRs.
• Build CheXagent with a vision encoder, a vision-language bridger, and a language decoder; train via four stages including adapting an LLM to clinical text.
• Stage 0: train a clinical LLM using PMC abstracts, MIMIC-IV radiology reports, discharge summaries, Wikipedia terms, and CheXinstruct data.
• Stage 1: train a CXR vision encoder with ITC and IC objectives on MIMIC-CXR, PadChest, and BIMCV-COVID-19 datasets.
• Stage 2: train a vision-language bridger while freezing the LLM and vision encoder to align image-text representations.
• Stage 3: instruction-tune the multimodal model on CheXinstruct tasks with next-word prediction objective focusing on answers.

Figure 1: Overview of the proposed pipeline: CheXinstruct is a curation of datasets for instruction-tuning across various CXR tasks, CheXagent is our clinical FM for CXR interpretation, and CheXbench is our comprehensive FM evaluation benchmark. Two example CXR interpretation tasks include local fin

Experimental results

Research questions

• RQ1 Can a large-scale instruction-tuning dataset enable robust CXR interpretation by a multimodal foundation model?
• RQ2 How does a vision-language FM pretrained on CXRs perform across core perception and text-generation tasks compared with general- and medical-domain FMs?
• RQ3 What are the fairness implications and potential biases of such a model across sex, race, and age?
• RQ4 Does the proposed CheXbench provide a reliable benchmarking framework for multimodal CXR interpretation tasks?
• RQ5 To what extent can radiologist-level quality be achieved in findings generation and summarization tasks?

Key findings

• CheXagent outperforms general-domain FMs by an average of 97.5% on image perception tasks across CheXbench axis 1.
• CheXagent outperforms medical-domain FMs by an average of 55.7% on image perception tasks across axis 1.
• On view classification, CheXagent shows large performance gains (near-perfect) over baselines on MIMIC-CXR and CheXpert datasets.
• On visual question answering, CheXagent achieves strong results and generalizes to hold-out datasets (SLAKE, Rad-Restruct).
• In text generation tasks, CheXagent demonstrates superior findings generation on private and MIMIC-CXR datasets and competitive summarization performance relative to larger LLMs.
• A radiologist reader study shows CheXagent is comparable to physicians for findings summarization and highlights gaps in findings generation, with qualitative insights for improvement.
• Fairness analyses reveal performance disparities across sex, race, and age, underscoring the need for diverse data and bias mitigation.

Figure 2: Collection of datasets and tasks comprising CheXinstruct (Left). The four-stage training process of CheXagent, starting from adapting a general LLM for clinical use, through training a CXR vision encoder and a vision-language bridger, to the final stage of instruction tuning on diverse CXR