[论文解读] Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations

Jungo Kasai, Yuhei Kasai+3 more|arXiv (Cornell University)|Mar 31, 2023

Artificial Intelligence in Healthcare and Education被引用 50

一句话总结

GPT-4 通过日本国家医师执业资格考试，表现优于 ChatGPT 与 GPT-3，但仍落后大多数医学生，并且由于日本分词造成的成本更高，以及禁用选项等限制。

ABSTRACT

As large language models (LLMs) gain popularity among speakers of diverse languages, we believe that it is crucial to benchmark them to better understand model behaviors, failures, and limitations in languages beyond English. In this work, we evaluate LLM APIs (ChatGPT, GPT-3, and GPT-4) on the Japanese national medical licensing examinations from the past five years, including the current year. Our team comprises native Japanese-speaking NLP researchers and a practicing cardiologist based in Japan. Our experiments show that GPT-4 outperforms ChatGPT and GPT-3 and passes all six years of the exams, highlighting LLMs' potential in a language that is typologically distant from English. However, our evaluation also exposes critical limitations of the current LLM APIs. First, LLMs sometimes select prohibited choices that should be strictly avoided in medical practice in Japan, such as suggesting euthanasia. Further, our analysis shows that the API costs are generally higher and the maximum context size is smaller for Japanese because of the way non-Latin scripts are currently tokenized in the pipeline. We release our benchmark as Igaku QA as well as all model outputs and exam metadata. We hope that our results and benchmark will spur progress on more diverse applications of LLMs. Our benchmark is available at https://github.com/jungokasai/IgakuQA.

研究动机与目标

• 评估大语言模型在2018–2023年间是否能够应对日本国家医师资格考试（NMPQE）。
• 比较 GPT-3、ChatGPT 和 GPT-4 在非英语、特定国家医考上的表现。
• 识别当前大语言模型 API 在日语医学领域中的领域与语言特定局限。
• 将 Igaku QA 作为原生日语基准，以促进多样化的非英语 大语言模型评估与发展。

提出的方法

• 在 Igaku QA 上以闭卷提示对 LLM API（GPT-3、ChatGPT、GPT-4）进行基准测试。
• 使用来自2006年日本医师执照考试的三个上下文内示例来提示模型回答。
• 由于MCQ格式和固定数值题，采用逐字匹配来评估答案。
• 包含 ChatGPT-EN 变体，将题目翻译成英文以测试多语言提示效果。
• 分析日本语环境下的禁忌肢、分词相关成本以及上下文窗口限制等因素。

Figure 1: Example problem from the Japanese medical licensing exam where ChatGPT chooses a prohibited choice (禁忌肢) because euthanasia is illegal in Japan . Test takers who choose four or more prohibited choices would fail regardless of their exam total scores (§ 2.2 ). The problem and the ChatGPT ou

实验结果

研究问题

• RQ1 GPT-4 能否在多年份（2018–2023）通过日本 NMPQE？
• RQ2 GPT-3、ChatGPT 与 GPT-4 如何相互比较以及相对于医学生在 Igaku QA 的表现？
• RQ3 将当前 LLM 应用于日语医学执照题时的关键局限是什么（如禁忌选项、地理/时间上下文、分词成本等）？

主要发现

• GPT-4 在所测模型中取得最佳表现。
• GPT-4 在所有六个年度（2018–2023）的日本医师执照考试中及格。
• GPT-4 明显低于医学生多数票的表现。
• 一些 LLM 输出偶尔选择禁忌肢（禁忌肢），在日本医疗实践中应避免。
• 日语分词增加了 API 成本并相对于英语减少了最大上下文窗口。
• ChatGPT-EN（英文提示/翻译）通常优于普通的 ChatGPT，表明在未进行显式翻译时存在多语言限制。

Figure 2: Standard timeline comparisons of the medical licensing processes in Japan and the United States. The Japanese system involves only one national licensing examination (NMPQE) towards the end of the six-year medical school education. The United States Medical Licensing Examination (USMLE) co