[论文解读] Domain-adapted large language models for classifying nuclear medicine reports

Zachary Huemann, Changhee Lee+3 more|arXiv (Cornell University)|Mar 1, 2023

Radiomics and Machine Learning in Medical ImagingMedicine被引用 3

一句话总结

本研究评估了领域自适应大语言模型在分类核医学报告中的表现，特别是从¹⁸F-FDG PET/CT报告中预测五分制Deauville评分。通过在核医学特定文本上对通用模型（如RoBERTa）进行掩码语言建模的微调，领域自适应使五分类准确率提升至77.4%——超过人类专家（66%），并优于仅视觉模型和多模态模型。

ABSTRACT

With the growing use of transformer-based language models in medicine, it is unclear how well these models generalize to nuclear medicine which has domain-specific vocabulary and unique reporting styles. In this study, we evaluated the value of domain adaptation in nuclear medicine by adapting language models for the purpose of 5-point Deauville score prediction based on clinical 18F-fluorodeoxyglucose (FDG) PET/CT reports. We retrospectively retrieved 4542 text reports and 1664 images for FDG PET/CT lymphoma exams from 2008-2018 in our clinical imaging database. Deauville scores were removed from the reports and then the remaining text in the reports was used as the model input. Multiple general-purpose transformer language models were used to classify the reports into Deauville scores 1-5. We then adapted the models to the nuclear medicine domain using masked language modeling and assessed its impact on classification performance. The language models were compared against vision models, a multimodal vision language model, and a nuclear medicine physician with seven-fold Monte Carlo cross validation, reported are the mean and standard deviations. Domain adaption improved all language models. For example, BERT improved from 61.3% five-class accuracy to 65.7% following domain adaptation. The best performing model (domain-adapted RoBERTa) achieved a five-class accuracy of 77.4%, which was better than the physician's performance (66%), the best vision model's performance (48.1), and was similar to the multimodal model's performance (77.2). Domain adaptation improved the performance of large language models in interpreting nuclear medicine text reports.

研究动机与目标

• 评估领域自适应是否能提升大语言模型在解读核医学报告方面的性能。
• 评估语言模型是否能基于自由文本PET/CT报告准确预测用于淋巴瘤反应评估的Deauville评分。
• 将领域自适应语言模型的性能与视觉模型、多模态模型及一名人类专家进行比较。
• 确定针对特定领域的预训练是否比通用生物医学领域或通用目的预训练更有效于核医学文本。
• 探究当报告和评分均由医生评定时，影像信息是否能为基于文本的分类带来额外价值。

提出的方法

• 从单一机构的PACS数据库（2008–2018年）中检索了4,542份FDG PET/CT报告和1,664张影像，Deauville评分通过报告文本的N-gram分析提取。
• 从报告中删除Deauville评分，以创建用于语言模型训练和推理的掩码输入文本。
• 利用核医学语料库上的自监督掩码语言建模方法，对通用Transformer模型（如BERT、RoBERTa）进行领域自适应。
• 使用七折蒙特卡洛交叉验证训练并评估多种语言模型、视觉模型（如ResNet）以及一个多模态视觉-语言模型。
• 以五分类准确率为首要指标，报告各折的平均值±标准差。
• 将领域自适应模型与非领域自适应版本、视觉模型、多模态模型以及一名核医学医生进行比较。

实验结果

研究问题

• RQ1 领域自适应是否能提升大语言模型在核医学报告分类以预测Deauville评分方面的性能？
• RQ2 领域自适应语言模型在该任务上的表现与视觉模型和多模态模型相比如何？
• RQ3 领域自适应语言模型能否在从文本报告预测Deauville评分方面超越人类专家？
• RQ4 与通用生物医学领域自适应（如BioClinicalBERT）或通用目的预训练（如BERT）相比，核医学领域自适应是否更有效？
• RQ5 当报告和评分均由医生评定时，PET/CT影像信息是否能为基于文本报告的Deauville评分预测带来额外增益？

主要发现

• 领域自适应显著提升了所有测试的语言模型性能，其中BERT的五分类准确率从61.3%提升至65.7%。
• 表现最佳的模型——领域自适应的RoBERTa——取得了77.4% ± 3.4%的五分类准确率，超过人类专家的66%。
• 多模态模型达到了相近的准确率77.2% ± 3.2%，表明语言信息在该预测任务中占主导地位。
• 仅视觉模型表现较差，最高准确率为48.1% ± 3.5%，表明仅依靠图像特征不足以完成此任务。
• RadBERT（一种在核医学文本上微调的小型模型）的性能与更大的RoBERTa模型相当，证明了领域特定预训练的价值。
• 通用目的和生物医学自适应语言模型（如BioClinicalBERT）并未优于领域自适应模型，表明核医学特定自适应至关重要。