[Paper Review] Is Open-Source There Yet? A Comparative Study on Commercial and Open-Source LLMs in Their Ability to Label Chest X-Ray Reports

Felix J. Dorfner, Liv Jürgensen+12 more|arXiv (Cornell University)|Feb 19, 2024

Radiomics and Machine Learning in Medical Imaging4 citations

TL;DR

This study compares commercial (GPT-3.5 Turbo, GPT-4) and open-source LLMs (Mistral-7B, Mixtral-8x7B, Llama2-13B/70B, QWEN1.5-72B, CheXbert, CheXpert-labeler) in labeling findings from chest X-ray reports using zero-shot and few-shot prompting. Open-source models, especially with few-shot prompting or ensemble methods, achieve performance nearly matching GPT-4, offering a cost-effective, privacy-preserving, and reproducible alternative for radiology report classification.

ABSTRACT

Introduction: With the rapid advances in large language models (LLMs), there have been numerous new open source as well as commercial models. While recent publications have explored GPT-4 in its application to extracting information of interest from radiology reports, there has not been a real-world comparison of GPT-4 to different leading open-source models. Materials and Methods: Two different and independent datasets were used. The first dataset consists of 540 chest x-ray reports that were created at the Massachusetts General Hospital between July 2019 and July 2021. The second dataset consists of 500 chest x-ray reports from the ImaGenome dataset. We then compared the commercial models GPT-3.5 Turbo and GPT-4 from OpenAI to the open-source models Mistral-7B, Mixtral-8x7B, Llama2-13B, Llama2-70B, QWEN1.5-72B and CheXbert and CheXpert-labeler in their ability to accurately label the presence of multiple findings in x-ray text reports using different prompting techniques. Results: On the ImaGenome dataset, the best performing open-source model was Llama2-70B with micro F1-scores of 0.972 and 0.970 for zero- and few-shot prompts, respectively. GPT-4 achieved micro F1-scores of 0.975 and 0.984, respectively. On the institutional dataset, the best performing open-source model was QWEN1.5-72B with micro F1-scores of 0.952 and 0.965 for zero- and few-shot prompting, respectively. GPT-4 achieved micro F1-scores of 0.975 and 0.973, respectively. Conclusion: In this paper, we show that while GPT-4 is superior to open-source models in zero-shot report labeling, the implementation of few-shot prompting can bring open-source models on par with GPT-4. This shows that open-source models could be a performant and privacy preserving alternative to GPT-4 for the task of radiology report classification.

Motivation & Objective

• To evaluate whether open-source LLMs can match the performance of commercial models like GPT-4 in labeling findings from chest X-ray reports.
• To assess the impact of prompting strategies—zero-shot versus few-shot—on model performance across diverse datasets.
• To investigate the feasibility of using open-source LLMs as a privacy-preserving, cost-effective, and reproducible alternative to proprietary models in clinical NLP.
• To explore ensemble methods combining top-performing open-source models to improve classification accuracy.
• To address challenges related to class imbalance and prompt optimization in medical report labeling tasks.

Proposed method

• Two independent datasets were used: 540 institutional chest X-ray reports from Massachusetts General Hospital and 500 reports from the ImaGenome dataset.
• Six open-source LLMs (Mistral-7B, Mixtral-8x7B, Llama2-13B, Llama2-70B, QWEN1.5-72B, CheXbert, CheXpert-labeler) and two commercial models (GPT-3.5 Turbo, GPT-4) were evaluated.
• Zero-shot and few-shot prompting techniques were applied, with prompts designed and tested initially on GPT-4 before adaptation to other models.
• Performance was measured using micro and macro F1-scores across 14 radiological findings, with class imbalance addressed via oversampling in the institutional dataset.
• An ensemble model was created by combining predictions from the three highest-performing open-source models (Mixtral-8x7B, Llama2-70B, QWEN1.5-72B) using majority voting.
• Model inference was conducted locally or via cloud providers, with quantization techniques enabling deployment on limited hardware (e.g., 48 GB VRAM for Llama-2 70B in Int4).

Figure 1: Flow-chart for the reports from the ImaGenome and the institutional dataset

Experimental results

Research questions

• RQ1 Can open-source LLMs achieve performance comparable to GPT-4 in classifying findings from unstructured chest X-ray reports?
• RQ2 How does few-shot prompting compare to zero-shot prompting in improving the accuracy of open-source LLMs for this task?
• RQ3 To what extent can ensemble methods combining multiple open-source models close the performance gap with GPT-4?
• RQ4 What are the practical advantages of open-source LLMs over commercial models in terms of cost, privacy, and reproducibility in medical AI?
• RQ5 How do class imbalance and prompt design affect model performance in radiology report classification?

Key findings

• On the ImaGenome dataset, the best open-source model, Llama2-70B, achieved micro F1-scores of 0.972 (zero-shot) and 0.970 (few-shot), compared to GPT-4’s 0.975 and 0.984, respectively.
• On the institutional dataset, QWEN1.5-72B achieved micro F1-scores of 0.952 (zero-shot) and 0.965 (few-shot), while GPT-4 achieved 0.975 and 0.973, respectively.
• An ensemble of Mixtral-8x7B, Llama2-70B, and QWEN1.5-72B matched GPT-4’s performance on the institutional dataset with few-shot prompting, achieving micro F1-scores of 0.971 and 0.973.
• The ensemble outperformed GPT-3.5 Turbo on both zero-shot and few-shot tasks across both datasets, demonstrating the value of model combination.
• Open-source models showed strong performance on frequently mentioned findings like 'Pneumothorax' and 'Atelectasis', where labels were often mentioned literally in reports.
• The study confirms that open-source LLMs offer significant advantages in cost, privacy, and reproducibility, especially when models are deployed locally and do not rely on external APIs.

Figure 2: Outline of the zero-shot prompting workflow for the ImaGenome dataset. The model is given instructions and the report to classify and outputs its response according to the JSON formatted template with the corresponding labels.