[论文解读] Radiology-Llama2: Best-in-Class Large Language Model for Radiology

Zhengliang Liu, Yiwei Li+19 more|arXiv (Cornell University)|Aug 29, 2023

Topic Modeling被引用 30

一句话总结

Radiology-Llama2 是一个基于 Llama2 的指令微调大型语言模型，专注于放射科报告以生成简明、临床有用的放射科印象，并且在 MIMIC-CXR 和 OpenI Rouge 指标上优于其他模型，得到专家支持。

ABSTRACT

This paper introduces Radiology-Llama2, a large language model specialized for radiology through a process known as instruction tuning. Radiology-Llama2 is based on the Llama2 architecture and further trained on a large dataset of radiology reports to generate coherent and clinically useful impressions from radiological findings. Quantitative evaluations using ROUGE metrics on the MIMIC-CXR and OpenI datasets demonstrate that Radiology-Llama2 achieves state-of-the-art performance compared to other generative language models, with a Rouge-1 score of 0.4834 on MIMIC-CXR and 0.4185 on OpenI. Additional assessments by radiology experts highlight the model's strengths in understandability, coherence, relevance, conciseness, and clinical utility. The work illustrates the potential of localized language models designed and tuned for specialized domains like radiology. When properly evaluated and deployed, such models can transform fields like radiology by automating rote tasks and enhancing human expertise.

研究动机与目标

• 由于通用模型在隐私和领域特定知识方面存在差距，推动在放射科本地化的 LLM 的需求。
• 描述指令微调作为将 LLM 与放射科特定任务（从发现到印象的转化）对齐的方法。
• 证明 Radiology-Llama2 在标准数据集上在放射科印象生成功能上优于其他模型。

提出的方法

• Base architecture: Llama2 with instruction tuning for radiology impressions.
• Dataset use: MIMIC-CXR and OpenI radiology reports with corresponding findings and impressions.
• Instruction tuning approach: format inputs as Findings -> Impression to align model outputs with clinical task.
• Training technique: LoRA-based fine-tuning with specified hyperparameters (lora_r=8, lora_alpha=16, lora_dropout=0.05).
• Evaluation: Rouge-1/2/L metrics and expert radiologist assessments on coherence, understandability, relevance, conciseness, and clinical utility.

Figure 1 : The overall framework of Radiology-Llama2.

实验结果

研究问题

• RQ1 Can a radiology-tuned LLM outperform general LLMs in generating concise and clinically useful radiology impressions?
• RQ2 Do domain-specific instruction tuning and datalead to improved coherence and utility of radiology reports across MIMIC-CXR and OpenI?
• RQ3 What is the comparative performance of Radiology-Llama2 against other radiology-focused models on standard Rouge metrics and expert assessments?

主要发现

• Radiology-Llama2 achieves state-of-the-art Rouge scores on MIMIC-CXR (ROUGE-1=0.4834, ROUGE-2=0.324, ROUGE-L=0.4427) and OpenI (ROUGE-1=0.4185, ROUGE-2=0.2569, ROUGE-L=0.4087).
• It outperforms Claude2, the second-best model, by large margins on Rouge metrics (e.g., MIMIC-CXR ROUGE-1 0.3177 vs 0.4834).
• Expert radiologist evaluations show Radiology-Llama2 ranking highest in understandability, coherence, conciseness, and clinical utility.
• Table-based results corroborate superior Rouge metrics across both datasets compared to multiple baselines.
• Radiology-Llama2 demonstrates robustness and generalizability across datasets and supports potential clinical utility and workflow integration.

Figure 2 : Performance of different LLMs on the Radiology task example.