[Paper Review] Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers

Israt Jahan, Md Tahmid Rahman Laskar+2 more|arXiv (Cornell University)|Jun 7, 2023

Topic Modeling4 citations

TL;DR

This study evaluates ChatGPT's zero-shot performance on biomedical NLP tasks—relation extraction, question answering, document classification, and summarization—comparing it against fine-tuned models like BioGPT and BioBART. Surprisingly, ChatGPT outperforms these specialized models on datasets with smaller training sets, demonstrating strong few-shot generalization in low-resource biomedical settings without fine-tuning.

ABSTRACT

ChatGPT is a large language model developed by OpenAI. Despite its impressive performance across various tasks, no prior work has investigated its capability in the biomedical domain yet. To this end, this paper aims to evaluate the performance of ChatGPT on various benchmark biomedical tasks, such as relation extraction, document classification, question answering, and summarization. To the best of our knowledge, this is the first work that conducts an extensive evaluation of ChatGPT in the biomedical domain. Interestingly, we find based on our evaluation that in biomedical datasets that have smaller training sets, zero-shot ChatGPT even outperforms the state-of-the-art fine-tuned generative transformer models, such as BioGPT and BioBART. This suggests that ChatGPT's pre-training on large text corpora makes it quite specialized even in the biomedical domain. Our findings demonstrate that ChatGPT has the potential to be a valuable tool for various tasks in the biomedical domain that lack large annotated data.

Motivation & Objective

• To evaluate the zero-shot performance of ChatGPT on benchmark biomedical NLP tasks without fine-tuning.
• To compare ChatGPT’s performance against state-of-the-art fine-tuned generative models like BioGPT and BioBART.
• To investigate whether large language models pre-trained on broad corpora can generalize effectively in low-resource biomedical settings.
• To assess the potential of zero-shot LLMs as viable alternatives when annotated biomedical data is scarce.
• To ensure reproducibility by releasing all ChatGPT-generated responses and evaluation code.

Proposed method

• Zero-shot inference using prompt engineering to guide ChatGPT on four biomedical tasks: relation extraction, question answering, document classification, and summarization.
• Prompts were carefully designed for each task, including instruction templates and few-shot examples where applicable.
• Evaluation was conducted on standard benchmark datasets: BC5CDR, KD-DTI, DDI, HoC, PubMedQA, iCliniq, HealthCare Magic, MeQSum, MEDIQA-QS, MEDIQA-MAS, MEDIQA-ANS.
• Performance was measured using standard metrics such as F1-score, accuracy, and ROUGE for summarization tasks.
• Fine-tuned models (BioGPT, BioBART) were used as baselines for direct comparison under identical evaluation conditions.
• All responses and evaluation code were released to ensure reproducibility and facilitate future research.

Experimental results

Research questions

• RQ1 Can ChatGPT perform biomedical NLP tasks effectively in a zero-shot setting without any fine-tuning?
• RQ2 How does ChatGPT’s zero-shot performance compare to state-of-the-art fine-tuned generative models like BioGPT and BioBART?
• RQ3 Does the performance of ChatGPT vary significantly across different biomedical tasks and dataset sizes?
• RQ4 In low-resource settings with small training sets, can a zero-shot LLM outperform fine-tuned models?
• RQ5 What is the potential of pre-trained LLMs like ChatGPT for biomedical applications where annotated data is limited?

Key findings

• ChatGPT achieved superior performance compared to fine-tuned models like BioGPT and BioBART on biomedical datasets with smaller training sets.
• In the BC5CDR dataset (500 train samples), ChatGPT outperformed fine-tuned models in chemical-disease relation extraction.
• For the HoC document classification task, ChatGPT demonstrated strong zero-shot accuracy despite no task-specific fine-tuning.
• On the PubMedQA question-answering benchmark, ChatGPT generated high-quality answers with strong factual consistency.
• In dialog summarization tasks (iCliniq and HealthCare Magic), ChatGPT produced coherent and informative summaries with competitive ROUGE scores.
• The results suggest that ChatGPT’s broad pre-training on diverse text corpora enables strong in-domain generalization even in specialized biomedical contexts.