[Paper Review] Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations

Jungo Kasai, Yuhei Kasai+3 more|arXiv (Cornell University)|Mar 31, 2023

Artificial Intelligence in Healthcare and EducationMedicine50 citations

TL;DR

GPT-4 passes Japan’s national medical licensing exams, outperforming ChatGPT and GPT-3, but still trails the majority of medical students and shows limitations like prohibited-choice selections and higher costs due to Japanese tokenization.

ABSTRACT

As large language models (LLMs) gain popularity among speakers of diverse languages, we believe that it is crucial to benchmark them to better understand model behaviors, failures, and limitations in languages beyond English. In this work, we evaluate LLM APIs (ChatGPT, GPT-3, and GPT-4) on the Japanese national medical licensing examinations from the past five years, including the current year. Our team comprises native Japanese-speaking NLP researchers and a practicing cardiologist based in Japan. Our experiments show that GPT-4 outperforms ChatGPT and GPT-3 and passes all six years of the exams, highlighting LLMs' potential in a language that is typologically distant from English. However, our evaluation also exposes critical limitations of the current LLM APIs. First, LLMs sometimes select prohibited choices that should be strictly avoided in medical practice in Japan, such as suggesting euthanasia. Further, our analysis shows that the API costs are generally higher and the maximum context size is smaller for Japanese because of the way non-Latin scripts are currently tokenized in the pipeline. We release our benchmark as Igaku QA as well as all model outputs and exam metadata. We hope that our results and benchmark will spur progress on more diverse applications of LLMs. Our benchmark is available at https://github.com/jungokasai/IgakuQA.

Motivation & Objective

• Assess whether large language models can handle the Japanese National Medical Practitioners Qualifying Examination (NMPQE) across 2018–2023.
• Compare GPT-3, ChatGPT, and GPT-4 performance on a non-English, country-specific medical exam.
• Identify domain- and language-specific limitations of current LLM APIs in Japanese medical contexts.
• Release Igaku QA as a native Japanese benchmark to spur diverse, non-English LLM evaluation and development.

Proposed method

• Benchmark LLM APIs (GPT-3, ChatGPT, GPT-4) on Igaku QA using closed-book prompts.
• Use three in-context examples from a 2006 Japanese medical licensing exam to cue model responses.
• Evaluate answers by exact matching due to MCQ format and fixed numerical questions.
• Include ChatGPT-EN variant that translates problems to English to test multilingual prompting effects.
• Analyze factors such as prohibited choices (禁忌肢), tokenization-related costs, and context window limits in Japanese.

Figure 1: Example problem from the Japanese medical licensing exam where ChatGPT chooses a prohibited choice (禁忌肢) because euthanasia is illegal in Japan . Test takers who choose four or more prohibited choices would fail regardless of their exam total scores (§ 2.2 ). The problem and the ChatGPT ou

Experimental results

Research questions

• RQ1 Can GPT-4 pass the Japanese NMPQE across multiple years (2018–2023)?
• RQ2 How do GPT-3, ChatGPT, and GPT-4 compare to each other and to medical student performance on Igaku QA?
• RQ3 What are the key limitations of current LLMs when applied to Japanese medical licensing questions (e.g., prohibited choices, geographic/temporal context, tokenization costs)?

Key findings

• GPT-4 achieves the best performance among the tested models.
• GPT-4 passes the Japanese medical licensing examinations across all six years (2018–2023).
• GPT-4 substantially underperforms the majority-vote performance of medical students.
• Some LLM outputs occasionally select prohibited choices (禁忌肢) that should be avoided in Japanese medical practice.
• Japanese tokenization increases API costs and reduces the maximum context window compared to English.
• ChatGPT-EN (English prompt/translation) often outperforms vanilla ChatGPT, indicating multilingual limitations without explicit translation.

Figure 2: Standard timeline comparisons of the medical licensing processes in Japan and the United States. The Japanese system involves only one national licensing examination (NMPQE) towards the end of the six-year medical school education. The United States Medical Licensing Examination (USMLE) co