[Paper Review] Performance of ChatGPT on USMLE: Unlocking the Potential of Large Language Models for AI-Assisted Medical Education

Prabin Sharma, Kisan Thapa+5 more|arXiv (Cornell University)|Jun 30, 2023

Artificial Intelligence in Healthcare and Education16 citations

TL;DR

The study evaluates ChatGPT on USMLE-style questions using Harvard anatomy data and physician adjudication, finding ChatGPT more context-oriented with deductive reasoning than Google, and achieving 58.8% on logical and 60% on ethical questions.

ABSTRACT

Artificial intelligence is gaining traction in more ways than ever before. The popularity of language models and AI-based businesses has soared since ChatGPT was made available to the general public via OpenAI. It is becoming increasingly common for people to use ChatGPT both professionally and personally. Considering the widespread use of ChatGPT and the reliance people place on it, this study determined how reliable ChatGPT can be for answering complex medical and clinical questions. Harvard University gross anatomy along with the United States Medical Licensing Examination (USMLE) questionnaire were used to accomplish the objective. The paper evaluated the obtained results using a 2-way ANOVA and posthoc analysis. Both showed systematic covariation between format and prompt. Furthermore, the physician adjudicators independently rated the outcome's accuracy, concordance, and insight. As a result of the analysis, ChatGPT-generated answers were found to be more context-oriented and represented a better model for deductive reasoning than regular Google search results. Furthermore, ChatGPT obtained 58.8% on logical questions and 60% on ethical questions. This means that the ChatGPT is approaching the passing range for logical questions and has crossed the threshold for ethical questions. The paper believes ChatGPT and other language learning models can be invaluable tools for e-learners; however, the study suggests that there is still room to improve their accuracy. In order to improve ChatGPT's performance in the future, further research is needed to better understand how it can answer different types of questions.

Motivation & Objective

• Assess ChatGPT's reliability for answering complex medical and clinical questions relevant to USMLE-style assessments.
• Compare ChatGPT performance to traditional search methods (Google) on medical question formats.
• Evaluate the influence of question format and prompting on ChatGPT performance using statistical analysis.
• Involve physician adjudicators to rate accuracy, concordance, and insight of AI-generated answers.

Proposed method

• Use Harvard University gross anatomy content and USMLE-style questions as evaluation materials.
• Apply a 2-way ANOVA to analyze performance with respect to format and prompt.
• Conduct post hoc analyses to explore interaction effects between format and prompt.
• Have physician adjudicators independently rate AI outputs for accuracy, concordance, and insight.

Experimental results

Research questions

• RQ1 Can ChatGPT reliably answer USMLE-style questions across logical and ethical domains?
• RQ2 Does question format or prompt style systematically affect ChatGPT performance?
• RQ3 How do physician evaluators rate the accuracy, concordance, and insight of ChatGPT responses compared to standard search results?

Key findings

• ChatGPT-generated answers were more context-oriented than Google search results.
• ChatGPT showed better deductive reasoning in responses than regular Google search results.
• ChatGPT scored 58.8% on logical questions and 60% on ethical questions.
• ChatGPT appears to be approaching passing range for logical questions and has crossed the threshold for ethical questions.
• Statistical analyses indicated systematic covariation between format and prompt, as shown by the 2-way ANOVA and post hoc analyses.