[Paper Review] Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine

Harsha Nori, Yin Tat Lee+16 more|arXiv (Cornell University)|Nov 28, 2023

Artificial Intelligence in Healthcare and EducationMedicine163 citations

TL;DR

The paper shows that prompt engineering (Medprompt) can unlock GPT-4’s specialist capabilities in medicine, surpassing state-of-the-art specialist models like Med-PaLM 2 across nine medical benchmarks with fewer model calls.

ABSTRACT

Generalist foundation models such as GPT-4 have displayed surprising capabilities in a wide variety of domains and tasks. Yet, there is a prevalent assumption that they cannot match specialist capabilities of fine-tuned models. For example, most explorations to date on medical competency benchmarks have leveraged domain-specific training, as exemplified by efforts on BioGPT and Med-PaLM. We build on a prior study of GPT-4's capabilities on medical challenge benchmarks in the absence of special training. Rather than using simple prompting to highlight the model's out-of-the-box capabilities, we perform a systematic exploration of prompt engineering. We find that prompting innovation can unlock deeper specialist capabilities and show that GPT-4 easily tops prior leading results for medical benchmarks. The prompting methods we explore are general purpose, and make no specific use of domain expertise, removing the need for expert-curated content. Our experimental design carefully controls for overfitting during the prompt engineering process. We introduce Medprompt, based on a composition of several prompting strategies. With Medprompt, GPT-4 achieves state-of-the-art results on all nine of the benchmark datasets in the MultiMedQA suite. The method outperforms leading specialist models such as Med-PaLM 2 by a significant margin with an order of magnitude fewer calls to the model. Steering GPT-4 with Medprompt achieves a 27% reduction in error rate on the MedQA dataset over the best methods to date achieved with specialist models and surpasses a score of 90% for the first time. Beyond medical problems, we show the power of Medprompt to generalize to other domains and provide evidence for the broad applicability of the approach via studies of the strategy on exams in electrical engineering, machine learning, philosophy, accounting, law, nursing, and clinical psychology.

Motivation & Objective

• Demonstrate that generalist foundation models can achieve or exceed specialist performance in medicine without extensive fine-tuning.
• Systematically explore prompt engineering techniques to unlock domain-specific capabilities.
• Develop and evaluate a general prompting framework (Medprompt) that is cost-effective and broadly applicable.

Proposed method

• Dynamic few-shot exemplar selection using k-NN in embedding space to choose relevant training examples.
• Self-generated chain-of-thought prompts produced automatically by GPT-4 with a verification step.
• Choice shuffling ensembling to reduce option position bias and improve robustness.
• Integration of the above into a two-stage Medprompt workflow with preprocessing and inference phases.
• A robust evaluation design with an eyes-off holdout dataset to guard against overfitting in prompting.
• Ablation studies to quantify the contribution of each Medprompt component.

(a)

Experimental results

Research questions

• RQ1 Can a generalist foundation model achieve state-of-the-art performance on medical challenge benchmarks without domain-specific fine-tuning?
• RQ2 Which prompting components contribute most to performance gains on medical QA tasks?
• RQ3 How well does Medprompt generalize to non-medical domains and competency exams?
• RQ4 What is the impact of prompt design on overfitting and generalization when evaluating on held-out data?

Key findings

• GPT-4 with Medprompt surpasses all baselines on nine MultiMedQA datasets, including MedQA, MedMCQA, PubMedQA, and MMLU medical subsets.
• On MedQA (USMLE-style), Medprompt achieves 90.6% accuracy under certain settings, a 27% reduction in error over prior best specialist methods.
• Self-generated chain-of-thought prompts contribute the largest single gain among Medprompt components, followed by few-shot exemplars and choice shuffle ensembling.
• Medprompt shows strong generalization to out-of-domain competency exams outside medicine, with average improvements of about 7.3 percentage points over zero-shot baselines across diverse domains.
• Eyes-off holdout tests indicate comparable or higher performance than eyes-on tests, suggesting low overfitting risk to benchmark prompts.
• Ablation indicates potential for further gains with more exemplars and ensemble steps, albeit with higher compute costs.

(b)