[Paper Review] MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosis

Asma Alkhaldi, Raneem Alnajim+6 more|arXiv (Cornell University)|Jul 4, 2024

Radiomics and Machine Learning in Medical Imaging5 citations

TL;DR

MiniGPT-Med is a vision-language model built on an LLM (LLaMA-2) with a frozen vision encoder (EVA) and a linear projection layer, enabling medical report generation, disease detection, and medical VQA across X-rays, CT, and MRIs, achieving state-of-the-art results in medical report generation and strong performance across grounding and non-grounding tasks.

ABSTRACT

Recent advancements in artificial intelligence (AI) have precipitated significant breakthroughs in healthcare, particularly in refining diagnostic procedures. However, previous studies have often been constrained to limited functionalities. This study introduces MiniGPT-Med, a vision-language model derived from large-scale language models and tailored for medical applications. MiniGPT-Med demonstrates remarkable versatility across various imaging modalities, including X-rays, CT scans, and MRIs, enhancing its utility. The model is capable of performing tasks such as medical report generation, visual question answering (VQA), and disease identification within medical imagery. Its integrated processing of both image and textual clinical data markedly improves diagnostic accuracy. Our empirical assessments confirm MiniGPT-Med's superior performance in disease grounding, medical report generation, and VQA benchmarks, representing a significant step towards reducing the gap in assisting radiology practice. Furthermore, it achieves state-of-the-art performance on medical report generation, higher than the previous best model by 19\% accuracy. MiniGPT-Med promises to become a general interface for radiology diagnoses, enhancing diagnostic efficiency across a wide range of medical imaging applications.

Motivation & Objective

• Motivate the need for a versatile radiology-focused multimodal model that can handle grounding and non-grounding tasks.
• Propose a unified architecture that leverages an LLM as a universal interface for medical vision-language tasks.
• Demonstrate state-of-the-art performance in medical report generation and competitive results in disease detection and VQA.
• Showcase the model’s applicability across X-rays, CT scans, and MRIs and its ability to generate detailed clinical reports.

Proposed method

• Use EVA as a frozen visual backbone to extract radiology image features.
• Adopt MiniGPT-v2 architecture with a linear projection layer to map visual tokens into the LLaMA-2 space.
• Incorporate task-specific prompts with [INST] tokens and six task identifiers to guide grounding and non-grounding tasks.
• Represent grounding bounding boxes textually as normalized coordinates in a [0,100] range for integration with the LLM.
• Fine-tune the LLaMA-2 model with LoRA while keeping the vision encoder frozen.
• Train on 124,276 medical images (448x448) without augmentation on an A100 GPU for 100 epochs, using cross-entropy loss with AdamW optimizer.

Figure 1: The diverse capabilities by MiniGPT-Med. It can perform disease detection, medical visual question answering, and medical report generation. MiniGPT-Med effectively works with a wide range of radiological data (X-rays, CT scans, and MRIs) and is adept at diagnosing many diseases.

Experimental results

Research questions

• RQ1 Can a single, unified model perform medical report generation, disease detection, and medical VQA across diverse radiology modalities?
• RQ2 Does integrating visual and textual clinical data via a unified LLM interface improve diagnostic accuracy and grounding ability compared to specialized or generalist baselines?
• RQ3 What is the state-of-the-art performance achievable in medical report generation on MIMIC-CXR with this architecture?
• RQ4 How well does the model generalize to zero-shot disease detection and VQA on external radiology benchmarks?

Key findings

• MiniGPT-Med achieves state-of-the-art performance in medical report generation on MIMIC-CXR, surpassing the best baseline by 19.0% in BERT-Sim and 5.2% in CheXbert-Sim.
• The model demonstrates competitive IoU for disease grounding on RSNA, with an IoU of 0.26, closer to specialist models.
• In medical VQA, MiniGPT-Med attains 0.58 on RadVQA, outperforming generalist models like MiniGPT-v2 and OpenFlamingo, and approaching the specialist MedVIN level (0.62).
• Radiologist expert evaluation indicates 76% of generated reports were deemed high quality, with 19% medium and 5% poor quality.
• Across report generation, disease detection, and VQA, MiniGPT-Med shows strong performance against both specialist and generalist baselines, indicating effective grounding and non-grounding task handling.

Figure 2: MiniGPT-Med Architecture Overview: The architecture comprises a vision encoder, a linear projection layer, and a large language model. It processes a single medical image, transforming it into visual semantic features via a pre-trained vision encoder. These features are concatenated into a