[Paper Review] Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification

Shan Chen, Yingya Li+5 more|arXiv (Cornell University)|Apr 5, 2023

Topic Modeling27 references19 citations

TL;DR

The study evaluates ChatGPT models (GPT-3.5, GPT-4) on biomedical classification and reasoning tasks using over 10,000 proxy samples, finding fine-tuning and traditional models often outperform prompting the largest LLMs. BoW with logistic regression matches complex prompting performance, and prompt engineering requires substantial effort.

ABSTRACT

Recent advances in large language models (LLMs) have shown impressive ability in biomedical question-answering, but have not been adequately investigated for more specific biomedical applications. This study investigates the performance of LLMs such as the ChatGPT family of models (GPT-3.5s, GPT-4) in biomedical tasks beyond question-answering. Because no patient data can be passed to the OpenAI API public interface, we evaluated model performance with over 10000 samples as proxies for two fundamental tasks in the clinical domain - classification and reasoning. The first task is classifying whether statements of clinical and policy recommendations in scientific literature constitute health advice. The second task is causal relation detection from the biomedical literature. We compared LLMs with simpler models, such as bag-of-words (BoW) with logistic regression, and fine-tuned BioBERT models. Despite the excitement around viral ChatGPT, we found that fine-tuning for two fundamental NLP tasks remained the best strategy. The simple BoW model performed on par with the most complex LLM prompting. Prompt engineering required significant investment.

Motivation & Objective

• Assess the capability of the ChatGPT family for biomedical reasoning beyond QA.
• Evaluate two fundamental clinical NLP tasks using proxy samples due to data privacy constraints.
• Compare LLM prompting with traditional models and fine-tuned baselines in biomedical tasks.
• Inform best-practice guidance for deploying LLMs in biomedical text tasks.

Proposed method

• Use over 10,000 samples as proxies for clinical classification and causal relation tasks.
• Two tasks: (1) classify whether statements in literature constitute health advice; (2) detect causal relations in biomedical literature.
• Compare ChatGPT variants (GPT-3.5s, GPT-4) with baseline models (BoW with logistic regression, fine-tuned BioBERT).
• Avoid passing patient data to public OpenAI API; rely on surrogate data for evaluation.
• Assess impact of prompt engineering on LLM performance.

Experimental results

Research questions

• RQ1 Can ChatGPT models accurately classify statements as health advice in scientific literature?
• RQ2 Can ChatGPT models detect causal relations in biomedical literature?
• RQ3 How do ChatGPT models compare to BoW logistic regression and fine-tuned BioBERT on these tasks?
• RQ4 What is the relative effort required for prompt engineering versus traditional model fine-tuning?

Key findings

• Fine-tuning for the two NLP tasks remains the best strategy among the compared approaches.
• BoW with logistic regression performs on par with the most complex LLM prompting strategies.
• Prompt engineering requires substantial investment to reach competitive performance.
• ChatGPT models without task-specific fine-tuning generally underperform the fine-tuned BioBERT baseline.
• Overall, classic NLP approaches can rival or exceed prompting-based LLM performance on these biomedical tasks.