[Paper Review] Domain-adapted large language models for classifying nuclear medicine reports

Zachary Huemann, Changhee Lee+3 more|arXiv (Cornell University)|Mar 1, 2023

Radiomics and Machine Learning in Medical ImagingMedicine3 citations

TL;DR

This study evaluates domain-adapted large language models for classifying nuclear medicine reports, specifically predicting 5-point Deauville scores from 18F-FDG PET/CT reports. By fine-tuning general-purpose models like RoBERTa on nuclear medicine-specific text via masked language modeling, domain adaptation boosted five-class accuracy to 77.4%—surpassing a human expert (66%) and outperforming vision-only and multimodal models.

ABSTRACT

With the growing use of transformer-based language models in medicine, it is unclear how well these models generalize to nuclear medicine which has domain-specific vocabulary and unique reporting styles. In this study, we evaluated the value of domain adaptation in nuclear medicine by adapting language models for the purpose of 5-point Deauville score prediction based on clinical 18F-fluorodeoxyglucose (FDG) PET/CT reports. We retrospectively retrieved 4542 text reports and 1664 images for FDG PET/CT lymphoma exams from 2008-2018 in our clinical imaging database. Deauville scores were removed from the reports and then the remaining text in the reports was used as the model input. Multiple general-purpose transformer language models were used to classify the reports into Deauville scores 1-5. We then adapted the models to the nuclear medicine domain using masked language modeling and assessed its impact on classification performance. The language models were compared against vision models, a multimodal vision language model, and a nuclear medicine physician with seven-fold Monte Carlo cross validation, reported are the mean and standard deviations. Domain adaption improved all language models. For example, BERT improved from 61.3% five-class accuracy to 65.7% following domain adaptation. The best performing model (domain-adapted RoBERTa) achieved a five-class accuracy of 77.4%, which was better than the physician's performance (66%), the best vision model's performance (48.1), and was similar to the multimodal model's performance (77.2). Domain adaptation improved the performance of large language models in interpreting nuclear medicine text reports.

Motivation & Objective

• To evaluate whether domain adaptation improves the performance of large language models in interpreting nuclear medicine reports.
• To assess if language models can accurately predict Deauville scores—used for lymphoma response assessment—based on free-text PET/CT reports.
• To compare the performance of domain-adapted language models against vision models, multimodal models, and a human expert.
• To determine whether domain-specific pretraining is more effective than general biomedical or general-purpose pretraining for nuclear medicine text.
• To investigate whether image information adds value to text-based classification when the report and score are physician-assigned.

Proposed method

• Retrieved 4,542 FDG PET/CT reports and 1,664 images from a single-institution PACS database (2008–2018), with Deauville scores extracted via N-gram analysis of report text.
• Redacted Deauville scores from reports to create masked input text for language model training and inference.
• Adapted general-purpose transformer models (e.g., BERT, RoBERTa) using self-supervised masked language modeling on the nuclear medicine corpus.
• Trained and evaluated multiple language models, vision models (e.g., ResNet), and a multimodal vision-language model using seven-fold Monte Carlo cross-validation.
• Used five-class accuracy as the primary metric, reporting mean ± standard deviation across folds.
• Compared domain-adapted models to non-domain-adapted versions, vision models, multimodal models, and a single nuclear medicine physician.

Experimental results

Research questions

• RQ1 Does domain adaptation improve the performance of large language models in classifying nuclear medicine reports for Deauville score prediction?
• RQ2 How does the performance of domain-adapted language models compare to vision models and multimodal models on the same task?
• RQ3 Can domain-adapted language models outperform a human expert in predicting Deauville scores from text reports?
• RQ4 Is domain adaptation to nuclear medicine more effective than general biomedical domain adaptation (e.g., BioClinicalBERT) or general-purpose pretraining (e.g., BERT)?
• RQ5 Does image information from PET/CT scans provide additional benefit when predicting Deauville scores from text reports?

Key findings

• Domain adaptation significantly improved all tested language models, with BERT’s five-class accuracy increasing from 61.3% to 65.7% after adaptation.
• The best-performing model, domain-adapted RoBERTa, achieved a five-class accuracy of 77.4% ± 3.4%, outperforming the human expert’s accuracy of 66%.
• The multimodal model achieved a comparable accuracy of 77.2% ± 3.2%, indicating that language information dominated the prediction task.
• Vision-only models performed poorly, with a maximum accuracy of 48.1% ± 3.5%, suggesting that image features alone are insufficient for this task.
• RadBERT, a smaller model fine-tuned on nuclear medicine text, achieved performance comparable to the larger RoBERTa model, demonstrating the value of domain-specific pretraining.
• General-purpose and biomedical-adapted language models (e.g., BioClinicalBERT) did not outperform domain-adapted models, indicating that nuclear medicine-specific adaptation is essential.