[Paper Review] Evaluating GPT-3.5 and GPT-4 Models on Brazilian University Admission Exams
The paper evaluates GPT-3.5 and GPT-4 on Brazilian ENEM exams in Portuguese, showing GPT-4 with Chain-of-Thought prompts achieves up to 87% accuracy on ENEM 2022, outperforming GPT-3.5 by about 11 points, with CoT enabling substantial gains in math and science questions.
The present study aims to explore the capabilities of Language Models (LMs) in tackling high-stakes multiple-choice tests, represented here by the Exame Nacional do Ensino Médio (ENEM), a multidisciplinary entrance examination widely adopted by Brazilian universities. This exam poses challenging tasks for LMs, since its questions may span into multiple fields of knowledge, requiring understanding of information from diverse domains. For instance, a question may require comprehension of both statistics and biology to be solved. This work analyzed responses generated by GPT-3.5 and GPT-4 models for questions presented in the 2009-2017 exams, as well as for questions of the 2022 exam, which were made public after the training of the models was completed. Furthermore, different prompt strategies were tested, including the use of Chain-of-Thought (CoT) prompts to generate explanations for answers. On the 2022 edition, the best-performing model, GPT-4 with CoT, achieved an accuracy of 87%, largely surpassing GPT-3.5 by 11 points. The code and data used on experiments are available at https://github.com/piresramon/gpt-4-enem.
Motivation & Objective
- Assess how state-of-the-art LMs perform on ENEM, a high-stakes, multi-domain exam in Brazilian Portuguese.
Proposed method
- Evaluate GPT-3.5 and GPT-4 variants via zero-shot, few-shot, and few-shot with Chain-of-Thought prompts.
Experimental results
Research questions
- RQ1How well do GPT-3.5 and GPT-4 perform on ENEM 2009-2017 and ENEM 2022 datasets?
- RQ2Does Chain-of-Thought prompting improve accuracy on multi-domain, Portuguese-language exam questions?
- RQ3Can prompts reveal or mitigate potential memorization of 2022 ENEM content by the models?
Key findings
- GPT-4 with CoT achieves 87% accuracy on ENEM 2022, outperforming GPT-3.5 by about 11 points.
Better researchstarts right now
From reading papers to final review, dramatically reduce your research time.
No credit card · Free plan available
This review was created by AI and reviewed by human editors.