[Paper Review] From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond

Harsha Nori, Naoto Usuyama+5 more|arXiv (Cornell University)|Nov 6, 2024

Biomedical and Engineering Education8 citations

TL;DR

The paper evaluates OpenAI's o1-preview on medical benchmarks, comparing it to Medprompt-enhanced GPT-4, and analyzes prompting strategies, reasoning tokens, and cost-performance tradeoffs for run-time inference in medical tasks.

ABSTRACT

Run-time steering strategies like Medprompt are valuable for guiding large language models (LLMs) to top performance on challenging tasks. Medprompt demonstrates that a general LLM can be focused to deliver state-of-the-art performance on specialized domains like medicine by using a prompt to elicit a run-time strategy involving chain of thought reasoning and ensembling. OpenAI's o1-preview model represents a new paradigm, where a model is designed to do run-time reasoning before generating final responses. We seek to understand the behavior of o1-preview on a diverse set of medical challenge problem benchmarks. Following on the Medprompt study with GPT-4, we systematically evaluate the o1-preview model across various medical benchmarks. Notably, even without prompting techniques, o1-preview largely outperforms the GPT-4 series with Medprompt. We further systematically study the efficacy of classic prompt engineering strategies, as represented by Medprompt, within the new paradigm of reasoning models. We found that few-shot prompting hinders o1's performance, suggesting that in-context learning may no longer be an effective steering approach for reasoning-native models. While ensembling remains viable, it is resource-intensive and requires careful cost-performance optimization. Our cost and accuracy analysis across run-time strategies reveals a Pareto frontier, with GPT-4o representing a more affordable option and o1-preview achieving state-of-the-art performance at higher cost. Although o1-preview offers top performance, GPT-4o with steering strategies like Medprompt retains value in specific contexts. Moreover, we note that the o1-preview model has reached near-saturation on many existing medical benchmarks, underscoring the need for new, challenging benchmarks. We close with reflections on general directions for inference-time computation with LLMs.

Motivation & Objective

• Assess o1-preview's performance on diverse medical benchmarks compared to GPT-4 with Medprompt.
• Investigate whether classic Medprompt prompting remains beneficial under a reasoning-native model.
• Analyze the impact of prompting strategies, reasoning token usage, and ensembling on performance and cost.
• Explore the existence of a cost-accuracy Pareto frontier among run-time strategies.
• Discuss implications for inference-time computation and future benchmark development in medicine.

Proposed method

• Systematically evaluate o1-preview across medical benchmarks including MedQA, MedMCQA, MMLU (Medical), NCLEX, and JMLE-2024.
• Compare o1-preview to GPT-4 and GPT-4o with and without Medprompt-style strategies.
• Examine prompting variants (zero-shot, few-shot, Medprompt components) and ensemble approaches.
• Analyze reasoning token usage and its effect on performance.
• Assess cost versus accuracy across run-time strategies using API token pricing.

Experimental results

Research questions

• RQ1 How does o1-preview perform on diverse medical benchmarks relative to GPT-4 with Medprompt prompting?
• RQ2 Do classic Medprompt prompting techniques provide benefits when used with a reasoning-native model like o1-preview?
• RQ3 What is the impact of reasoning tokens and ensembling on accuracy and cost in run-time strategies?
• RQ4 Is there a cost-accuracy Pareto frontier among run-time strategies for medical benchmarks?
• RQ5 What are the implications for inference-time computation and benchmark development in medical AI?

Key findings

• o1-preview often outperforms GPT-4 guided by Medprompt on several medical benchmarks, even with simple prompting.
• Few-shot prompting tends to hurt o1-preview performance, while ensembling provides a consistent accuracy boost at higher cost.
• More reasoning tokens generally correlate with higher accuracy for o1-preview, and explicit CoT prompting is less recommended.
• GPT-4o offers a favorable cost-accuracy balance and can outperform older Medprompt configurations on many tasks.
• The o1-preview model demonstrates strong non-English medical reasoning on JMLE-2024, with run-time strategies further boosting results.
• Benchmarks show near-saturation on existing medical benchmarks, highlighting the need for new, challenging tasks.