[论文解读] ChatRadio-Valuer: A Chat Large Language Model for Generalizable Radiology Report Generation Based on Multi-institution and Multi-system Data

Tianyang Zhong, Wei Zhao+40 more|arXiv (Cornell University)|Oct 8, 2023

Artificial Intelligence in Healthcare and Education被引用 12

一句话总结

ChatRadio-Valuer 在来自多机构和多系统数据的 332,673 份放射科报告上对 Llama2 进行微调，以实现放射科报告生成的泛化，并在放射科报告的疾病诊断方面超越 ChatGPT 和 GPT-4。

ABSTRACT

Radiology report generation, as a key step in medical image analysis, is critical to the quantitative analysis of clinically informed decision-making levels. However, complex and diverse radiology reports with cross-source heterogeneity pose a huge generalizability challenge to the current methods under massive data volume, mainly because the style and normativity of radiology reports are obviously distinctive among institutions, body regions inspected and radiologists. Recently, the advent of large language models (LLM) offers great potential for recognizing signs of health conditions. To resolve the above problem, we collaborate with the Second Xiangya Hospital in China and propose ChatRadio-Valuer based on the LLM, a tailored model for automatic radiology report generation that learns generalizable representations and provides a basis pattern for model adaptation in sophisticated analysts' cases. Specifically, ChatRadio-Valuer is trained based on the radiology reports from a single institution by means of supervised fine-tuning, and then adapted to disease diagnosis tasks for human multi-system evaluation (i.e., chest, abdomen, muscle-skeleton, head, and maxillofacial $\&$ neck) from six different institutions in clinical-level events. The clinical dataset utilized in this study encompasses a remarkable total of extbf{332,673} observations. From the comprehensive results on engineering indicators, clinical efficacy and deployment cost metrics, it can be shown that ChatRadio-Valuer consistently outperforms state-of-the-art models, especially ChatGPT (GPT-3.5-Turbo) and GPT-4 et al., in terms of the diseases diagnosis from radiology reports. ChatRadio-Valuer provides an effective avenue to boost model generalization performance and alleviate the annotation workload of experts to enable the promotion of clinical AI applications in radiology reports.

研究动机与目标

• 开发一种完整且临床可用的放射科报告生成解决方案，使其在多家机构和多系统的范围内具有泛化能力。
• 使用单机构微调样本实现跨机构自适应放射科报告生成。
• 评估在六家机构和五个身体系统上的泛化能力。
• 评估临床效用和部署成本，以促进真实世界放射科 AI 的应用。

提出的方法

• 在大型放射科报告语料库上微调 Llama2，以学习可泛化的放射科知识。
• 通过专家驱动的清洗、提示综合和多系统/机构整合对数据进行预处理，以生成高质量的提示。
• 构建80/20的训练/评估分割，机构1的数据用于微调，其他机构用于测试。
• 通过将发现输入到 LLM 来生成放射科报告印象并提取印象。
• 使用工程度量和专家驱动的临床效用评估进行比较，以与最先进模型比较。

Figure 3 : The architecture diagram of Llama 2. The model structure of Llama 2 is basically consistent with the standard Transformer Decoder structure, mainly composed of 32 Transformer Blocks

实验结果

研究问题

• RQ1 ChatRadio-Valuer 是否能够在六家机构和五个放射系统上实现跨机构泛化？
• RQ2 ChatRadio-Valuer 在放射科报告生成和来自报告的疾病诊断方面，与最先进的模型（如 ChatGPT、GPT-4）相比如何？
• RQ3 该方法对标注工作量和实际临床效用有何影响？
• RQ4 哪些数据预处理和提示策略对实现跨异质放射数据的鲁棒泛化至关重要？

主要发现

• ChatRadio-Valuer 在放射科报告中的疾病诊断方面始终优于最先进的模型。
• 该框架在六家机构和五个系统上展示了跨机构和多系统的泛化。
• 数据预处理和专家策划的提示可降低噪声并提高手工提示质量，以实现鲁棒的微调。
• 该方法支持临床有效性评估和部署成本考量，有助于真实世界放射科 AI 部署。
• 该模型利用 Llama2 架构，具备上下文长度 4096，FFN 中的 SwiGLU，RMSNorm，RoPE，以及分组查询注意力，以处理异质放射科数据。

Figure 4 : Prompt generation overview. The overall framework contains three parts, system description, instruction, and input, which collaboratively constitute a prompt. Within a prompt example (purple), expert instruction and input data on its right are inserted to the { Expert Instruction } and {