[论文解读] Evaluating GPT-3.5 and GPT-4 Models on Brazilian University Admission Exams

Desnes Nunes, Ricardo Primi+3 more|arXiv (Cornell University)|Mar 29, 2023

Artificial Intelligence in Healthcare and Education被引用 24

一句话总结

本文评估 GPT-3.5 与 GPT-4 在葡萄牙语巴西 ENEM 考试上的表现，显示带有 Chain-of-Thought 提示的 GPT-4 在 ENEM 2022 上的准确率最高可达 87%，比 GPT-3.5 高出约 11 个点，CoT 使数学与科学题目获得显著提升。

ABSTRACT

The present study aims to explore the capabilities of Language Models (LMs) in tackling high-stakes multiple-choice tests, represented here by the Exame Nacional do Ensino Médio (ENEM), a multidisciplinary entrance examination widely adopted by Brazilian universities. This exam poses challenging tasks for LMs, since its questions may span into multiple fields of knowledge, requiring understanding of information from diverse domains. For instance, a question may require comprehension of both statistics and biology to be solved. This work analyzed responses generated by GPT-3.5 and GPT-4 models for questions presented in the 2009-2017 exams, as well as for questions of the 2022 exam, which were made public after the training of the models was completed. Furthermore, different prompt strategies were tested, including the use of Chain-of-Thought (CoT) prompts to generate explanations for answers. On the 2022 edition, the best-performing model, GPT-4 with CoT, achieved an accuracy of 87%, largely surpassing GPT-3.5 by 11 points. The code and data used on experiments are available at https://github.com/piresramon/gpt-4-enem.

研究动机与目标

• 评估最先进的语言模型在 ENEM 上的表现，ENEM 是一个以巴西葡萄牙语进行的高风险、覆盖多个领域的考试。

提出的方法

• 通过零样本、少量样本，以及带 Chain-of-Thought 提示的少量样本等变体来评估 GPT-3.5 和 GPT-4。

(a) Question 134 of Natural Sciences

实验结果

研究问题

• RQ1 GPT-3.5 和 GPT-4 在 ENEM 2009-2017 与 ENEM 2022 数据集上的表现如何？
• RQ2 Chain-of-Thought 提示是否提高了多领域、葡萄牙语题目上的准确率？
• RQ3 提示能否揭示或缓解模型对 2022 ENEM 内容的潜在记忆？

主要发现

• 带有 CoT 的 GPT-4 在 ENEM 2022 上的准确率达到 87%，比 GPT-3.5 高出约 11 点。

(b) Question 141 of Mathematics