[Paper Review] A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?

Yunfei Xie, Jianzhen Wu+7 more|arXiv (Cornell University)|Sep 23, 2024

Artificial Intelligence in Healthcare and Education14 citations

TL;DR

This paper evaluates OpenAI's o1 model on 37 medical datasets across understanding, reasoning, and multilinguality, showing improved medical understanding and reasoning but highlighting hallucination, multilingual challenges, and metric inconsistencies.

ABSTRACT

Large language models (LLMs) have exhibited remarkable capabilities across various domains and tasks, pushing the boundaries of our knowledge in learning and cognition. The latest model, OpenAI's o1, stands out as the first LLM with an internalized chain-of-thought technique using reinforcement learning strategies. While it has demonstrated surprisingly strong capabilities on various general language tasks, its performance in specialized fields such as medicine remains unknown. To this end, this report provides a comprehensive exploration of o1 on different medical scenarios, examining 3 key aspects: understanding, reasoning, and multilinguality. Specifically, our evaluation encompasses 6 tasks using data from 37 medical datasets, including two newly constructed and more challenging question-answering (QA) tasks based on professional medical quizzes from the New England Journal of Medicine (NEJM) and The Lancet. These datasets offer greater clinical relevance compared to standard medical QA benchmarks such as MedQA, translating more effectively into real-world clinical utility. Our analysis of o1 suggests that the enhanced reasoning ability of LLMs may (significantly) benefit their capability to understand various medical instructions and reason through complex clinical scenarios. Notably, o1 surpasses the previous GPT-4 in accuracy by an average of 6.2% and 6.6% across 19 datasets and two newly created complex QA scenarios. But meanwhile, we identify several weaknesses in both the model capability and the existing evaluation protocols, including hallucination, inconsistent multilingual ability, and discrepant metrics for evaluation. We release our raw data and model outputs at https://ucsc-vlaa.github.io/o1_medicine/ for future research.

Motivation & Objective

• Assess whether the o1 model's enhanced reasoning transfers to medical domains.
• Evaluate o1 on medical understanding, reasoning, and multilingual capabilities using diverse datasets.
• Compare o1 against GPT-4, GPT-3.5, and open-source baselines across multiple medical tasks.
• Identify weaknesses in model performance and current evaluation protocols to guide future clinical AI development.

Proposed method

• Assemble a broad evaluation suite of 37 datasets (35 existing + 2 novel) spanning three medical aspects.
• Use three prompting strategies: direct prompting, chain-of-thought (CoT), and few-shot prompting; assess impact of CoT given o1's internal CoT training.
• Compare o1 to GPT-4, GPT-3.5, MEDITRON-70B, and Llama3-8B across six tasks and three aspects.
• Employ metrics including Accuracy, F1, BLEU, ROUGE, AlignScore, and Mauve to evaluate different task types (understanding, reasoning, multilinguality).
• Analyze results with additional prompts (CoT, Self-Consistency, Reflex) to study prompting effects.

Experimental results

Research questions

• RQ1 Can o1’s internal chain-of-thought and reinforcement learning training improve clinical understanding and reasoning compared to prior models?
• RQ2 How does o1 perform across medical understanding, reasoning, and multilingual tasks relative to GPT-4, GPT-3.5, and open-source baselines?
• RQ3 What are the limitations of o1 in medical contexts, including hallucination and multilingual challenges, and how do evaluation metrics affect model ranking?

Key findings

• o1 generally outperforms GPT-4 and GPT-3.5 on understanding and certain reasoning tasks across many datasets.
• In new NEJMQA and LancetQA tasks, o1 shows notable accuracy gains over GPT-4 and GPT-3.5.
• o1 achieves higher ROUGE-1 scores in free-form generation tasks and demonstrates improved summarization quality.
• Hallucination remains a challenge for o1, and multilingual complex scenarios reveal gaps in multilingual reasoning.
• Evaluation metrics yield inconsistent rankings across models, highlighting the need for robust, domain-specific metrics.
• CoT prompting can improve o1’s medical knowledge tasks, though not universally across all task types.