[Paper Review] MGH Radiology Llama: A Llama 3 70B Model for Radiology

Yuan Shi, Peng Shu+6 more|arXiv (Cornell University)|Aug 13, 2024

Radiology practices and education4 citations

TL;DR

This paper introduces MGH Radiology Llama, a 70B-parameter Llama 3-based model fine-tuned on 6.5 million de-identified radiology reports from Massachusetts General Hospital. It achieves state-of-the-art performance in generating accurate, clinically relevant radiology impressions, outperforming general-purpose LLMs and prior domain-specific models, with a 100% ROUGE-L improvement and over 1-point gain on GPT-4o evaluation.

ABSTRACT

In recent years, the field of radiology has increasingly harnessed the power of artificial intelligence (AI) to enhance diagnostic accuracy, streamline workflows, and improve patient care. Large language models (LLMs) have emerged as particularly promising tools, offering significant potential in assisting radiologists with report generation, clinical decision support, and patient communication. This paper presents an advanced radiology-focused large language model: MGH Radiology Llama. It is developed using the Llama 3 70B model, building upon previous domain-specific models like Radiology-GPT and Radiology-Llama2. Leveraging a unique and comprehensive dataset from Massachusetts General Hospital, comprising over 6.5 million de-identified medical reports across various imaging modalities, the model demonstrates significant improvements in generating accurate and clinically relevant radiology impressions given the corresponding findings. Our evaluation, incorporating both traditional metrics and a GPT-4-based assessment, highlights the enhanced performance of this work over general-purpose LLMs.

Motivation & Objective

• To develop a high-performance, domain-specific large language model tailored for radiology report generation using a large-scale, real-world dataset.
• To address limitations of general-purpose LLMs in radiology, including lack of clinical precision, privacy concerns from API usage, and inconsistent medical terminology.
• To evaluate the model’s performance using both traditional metrics and GPT-4o-based assessment to ensure clinical relevance and accuracy.
• To explore efficient fine-tuning strategies like QLoRA for large models, balancing performance and computational efficiency.
• To maintain strict patient privacy through on-premise deployment and use of de-identified data.

Proposed method

• Fine-tuned the Llama 3 70B foundation model on a comprehensive dataset of 6.5 million de-identified radiology reports from MGH, covering multiple imaging modalities and anatomical regions.
• Employed two fine-tuning strategies: full fine-tuning and parameter-efficient LoRA (QLoRA), to compare performance and training efficiency.
• Preprocessed the dataset using domain-specific tokenization and data cleaning to ensure consistency and clinical relevance.
• Conducted evaluation using standard metrics (ROUGE-L, BERTScore) and a GPT-4o-based assessment with clinician-curated criteria for accuracy and semantic alignment.
• Used a prompt-based evaluation framework to assess the model’s ability to generate impressions that match radiologist-verified conclusions.
• Prioritized model privacy and compliance by training locally on de-identified data, avoiding external API calls.

Experimental results

Research questions

• RQ1 Can a large-scale, domain-specific LLM fine-tuned on a diverse, real-world radiology dataset significantly improve the accuracy and clinical relevance of radiology impression generation compared to general-purpose LLMs?
• RQ2 How does the performance of a fully fine-tuned Llama 3 70B model compare to a QLoRA-fine-tuned variant in generating radiology reports, particularly in terms of output quality and training efficiency?
• RQ3 To what extent can a large foundation model like Llama 3 70B leverage parameter-efficient fine-tuning (QLoRA) to achieve performance close to full fine-tuning while reducing computational cost?
• RQ4 What are the primary failure modes in generated radiology impressions, and how do they relate to missing findings or hallucinated conclusions?
• RQ5 How does the model’s performance vary across different imaging modalities and anatomical regions, and what does this suggest about its generalization capability?

Key findings

• The fully fine-tuned Llama 3 70B model achieved a 100% improvement in ROUGE-L score compared to the baseline general-purpose LLM, indicating superior word-level and structural alignment with ground-truth radiology impressions.
• The model improved BERTScore precision by 4%–5% over the baseline, demonstrating enhanced semantic similarity and medical terminology accuracy.
• The GPT-4o evaluation showed a 1.0+ point increase in average score for the fine-tuned models, indicating better alignment with radiologist-verified conclusions across clinical reasoning and terminology.
• QLoRA fine-tuning achieved performance nearly identical to full fine-tuning, with a significant reduction in training time and resource consumption, highlighting its efficiency for large models.
• Despite strong performance, the model still exhibits hallucination and missing key findings in complex cases, with low GPT-4o scores primarily due to omissions or irrelevant inferences.
• The model’s performance is robust across diverse imaging modalities (CT, MRI, X-ray, fluoroscopy) and anatomical regions (abdomen, chest, cardiac, limbs), indicating broad generalization capability.