[Paper Review] A Comparative Study of Open-Source Large Language Models, GPT-4 and Claude 2: Multiple-Choice Test Taking in Nephrology

Sean M. Wu, Michael Koo+5 more|arXiv (Cornell University)|Aug 9, 2023

Artificial Intelligence in Healthcare and Education24 citations

TL;DR

The paper compares open-source LLMs to GPT-4 and Claude 2 on nephrology multiple-choice questions, showing GPT-4 and Claude 2 outperform open-source models by large margins.

ABSTRACT

In recent years, there have been significant breakthroughs in the field of natural language processing, particularly with the development of large language models (LLMs). These LLMs have showcased remarkable capabilities on various benchmarks. In the healthcare field, the exact role LLMs and other future AI models will play remains unclear. There is a potential for these models in the future to be used as part of adaptive physician training, medical co-pilot applications, and digital patient interaction scenarios. The ability of AI models to participate in medical training and patient care will depend in part on their mastery of the knowledge content of specific medical fields. This study investigated the medical knowledge capability of LLMs, specifically in the context of internal medicine subspecialty multiple-choice test-taking ability. We compared the performance of several open-source LLMs (Koala 7B, Falcon 7B, Stable-Vicuna 13B, and Orca Mini 13B), to GPT-4 and Claude 2 on multiple-choice questions in the field of Nephrology. Nephrology was chosen as an example of a particularly conceptually complex subspecialty field within internal medicine. The study was conducted to evaluate the ability of LLM models to provide correct answers to nephSAP (Nephrology Self-Assessment Program) multiple-choice questions. The overall success of open-sourced LLMs in answering the 858 nephSAP multiple-choice questions correctly was 17.1% - 25.5%. In contrast, Claude 2 answered 54.4% of the questions correctly, whereas GPT-4 achieved a score of 73.3%. We show that current widely used open-sourced LLMs do poorly in their ability for zero-shot reasoning when compared to GPT-4 and Claude 2. The findings of this study potentially have significant implications for the future of subspecialty medical training and patient care.

Motivation & Objective

• Assess the nephrology knowledge abilities of open-source LLMs on nephSAP MCQs.
• Compare open-source LLMs with GPT-4 and Claude 2 in zero-shot performance on a nephrology dataset.
• Analyze the quality of model explanations and semantic alignment with ground truth.

Proposed method

• Evaluate Koala 7B, Falcon 7B, Stable-Vicuna 13B, and Orca Mini 13B against GPT-4 and Claude 2 on 858 nephSAP MCQs.
• Construct input prompts by concatenating Context, Question, and Choices for forward passes on open-source LLMs.
• Parse automated outputs using regex-based extraction to determine predicted answers and compare to ground truth.
• Measure accuracy as raw correct responses and compute topic-wise performance across nephrology subtopics.
• Assess explanations using BLEU, WER, and cosine similarity against ground-truth explanations.

Experimental results

Research questions

• RQ1 How do open-source LLMs perform on nephSAP nephrology MCQs compared with GPT-4 and Claude 2?
• RQ2 What are the topic-wise strengths and weaknesses of open-source LLMs in nephrology?
• RQ3 How do open-source LLMs compare in explanation quality to ground truth?
• RQ4 What factors might explain performance gaps between open-source LLMs and GPT-4/Claude 2?

Key findings

• GPT-4 achieved 629 correct answers (73.3%), Claude 2 467 (54.4%), and open-source LLMs ranged from 17.1% to 25.5% correct.
• Among open-source models, Vicuna had the highest score at 219 correct (25.5%), Koala followed with 204 correct (23.8%).
• Falcon scored 155 correct (18.1%), Orca-Mini 147 correct (17.1%), and Koala 204 correct (23.8%).
• Random guessing would yield 23.8% given the question structure, so only Koala slightly exceeded random expectation among open-source models.
• Explanations by open-source models showed low BLEU scores (e.g., Vicuna 9%, Falcon 8%, Orca-Mini 5%, Koala 5%), and cosine similarity scores were also sub-optimal across models.
• GPT-4 and Claude 2 outperformed open-source models across all nephrology topics, with GPT-4 showing near-human performance in most topics.