[Paper Review] Enhancing disease detection in radiology reports through fine-tuning lightweight LLM on weak labels

Yishu Wei, Xindi Wang+5 more|PubMed|Sep 25, 2024

Computational and Text Analysis Methods6 references4 citations

TL;DR

This paper proposes fine-tuning a lightweight LLM (Llama 3.1-8B) on weakly labeled radiology reports to enhance disease detection. By jointly training on multiple-choice classification and open-ended detection tasks using synthetic labels from GPT-4o or rule-based systems, the model achieves a micro-F1 of 0.91 on open-ended detection and surpasses its noisy teacher model, demonstrating strong potential for low-resource, privacy-preserving medical LLM specialization.

ABSTRACT

Despite significant progress in applying large language models (LLMs) to the medical domain, several limitations still prevent them from practical applications. Among these are the constraints on model size and the lack of cohort-specific labeled datasets. In this work, we investigated the potential of improving a lightweight LLM, such as Llama 3.1-8B, through fine-tuning with datasets using synthetic labels. Two tasks are jointly trained by combining their respective instruction datasets. When the quality of the task-specific synthetic labels is relatively high (e.g., generated by GPT4-o), Llama 3.1-8B achieves satisfactory performance on the open-ended disease detection task, with a micro F1 score of 0.91. Conversely, when the quality of the task-relevant synthetic labels is relatively low (e.g., from the MIMIC-CXR dataset), fine-tuned Llama 3.1-8B is able to surpass its noisy teacher labels (micro F1 score of 0.67 v.s. 0.63) when calibrated against curated labels, indicating the strong inherent underlying capability of the model. These findings demonstrate the potential offine-tuning LLMs with synthetic labels, offering a promising direction for future research on LLM specialization in the medical domain.

Motivation & Objective

• Address the challenge of limited, low-quality labeled data in radiology by leveraging synthetic labels for model fine-tuning.
• Overcome constraints of large LLMs in hospitals by using lightweight models like Llama 3.1-8B.
• Evaluate whether weakly supervised fine-tuning can match or exceed performance of human-annotated or rule-based labels.
• Investigate joint vs. separate fine-tuning of a single LLM on multiple radiology tasks.
• Demonstrate the feasibility of using GPT-4o-generated synthetic labels to train a lightweight model to near-expert performance in disease detection.

Proposed method

• Fine-tuned Llama 3.1-8B using instruction tuning on two distinct radiology tasks: multiple-choice disease classification and open-ended disease detection.
• Generated synthetic labels for the multiple-choice task using Negbio, a rule-based system, and for the open-ended task using GPT-4o.
• Combined instruction datasets from different sources (MIMIC-CXR and WCM) into a single mixed training set for joint multi-task learning.
• Applied instruction tuning with carefully engineered prompts that included negation handling, certainty filtering, and evidence extraction.
• Evaluated model performance using micro-F1 scores on curated human-annotated test sets, comparing against both rule-based labels and GPT-4o outputs.
• Explored both joint and separate fine-tuning strategies to assess performance trade-offs across tasks.

Experimental results

Research questions

• RQ1 Can a lightweight LLM like Llama 3.1-8B achieve high performance in radiology disease detection when fine-tuned on synthetic labels?
• RQ2 Does joint fine-tuning on multiple tasks improve model generalization compared to separate training?
• RQ3 Can a student model outperform its teacher model when the teacher uses noisy or imperfect labels?
• RQ4 How does the quality of synthetic labels (e.g., from GPT-4o vs. rule-based systems) affect downstream model performance?
• RQ5 To what extent can synthetic labels enable effective specialization of LLMs in low-resource, privacy-sensitive medical settings?

Key findings

• Llama 3.1-8B achieved a micro-F1 score of 0.91 on the open-ended disease detection task when fine-tuned on GPT-4o-generated synthetic labels.
• The fine-tuned Llama 3.1-8B outperformed its rule-based teacher (NegBio), achieving a micro-F1 of 0.66 compared to the teacher’s 0.63 on the multiple-choice classification task.
• When trained on low-quality synthetic labels from MIMIC-CXR, the fine-tuned Llama 3.1-8B still surpassed the performance of the noisy teacher model, indicating strong model generalization.
• Joint fine-tuning on both tasks yielded performance comparable to separate training, suggesting no significant degradation from multi-task learning.
• The model demonstrated robustness to label noise, achieving high performance even when trained on imperfect synthetic labels, highlighting its inherent capability for medical NLP tasks.
• The results indicate that synthetic labels from high-quality LLMs like GPT-4o can effectively transfer expert-level knowledge to lightweight models, enabling practical deployment in clinical settings.