[Paper Review] Exploring Durham University Physics exams with Large Language Models

Will Yeadon, D. P. Halliday|arXiv (Cornell University)|Jun 27, 2023

Artificial Intelligence in Healthcare and Education8 citations

TL;DR

GPT-4 and GPT-3.5 were evaluated on 42 Durham University physics exams (593 questions, 2504 marks) to assess AI capability and exam integrity; GPT-4 averaged 49.4% and GPT-3.5 averaged 38.6%, with modest post-COVID declines.

ABSTRACT

The emergence of advanced Natural Language Processing (NLP) models like ChatGPT has raised concerns among universities regarding AI-driven exam completion. This paper provides a comprehensive evaluation of the proficiency of GPT-4 and GPT-3.5 in answering a set of 42 exam papers derived from 10 distinct physics courses, administered at Durham University over the span of 2018 to 2022, totalling 593 questions and 2504 available marks. These exams, spanning both undergraduate and postgraduate levels, include traditional pre-COVID and adaptive COVID-era formats. Questions from the years 2018-2020 were designed for pre-COVID in person adjudicated examinations whereas the 2021-2022 exams were set for varying COVID-adapted conditions including open-book conditions. To ensure a fair evaluation of AI performances, the exams completed by AI were assessed by the original exam markers. However, due to staffing constraints, only the aforementioned 593 out of the total 1280 questions were marked. GPT-4 and GPT-3.5 scored an average of 49.4\% and 38.6\%, respectively, suggesting only the weaker students would potential improve their marks if using AI. For exams from the pre-COVID era, the average scores for GPT-4 and GPT-3.5 were 50.8\% and 41.6\%, respectively. However, post-COVID, these dropped to 47.5\% and 33.6\%. Thus contrary to expectations, the change to less fact-based questions in the COVID era did not significantly impact AI performance for the state-of-the-art models such as GPT-4. These findings suggest that while current AI models struggle with university-level Physics questions, an improving trend is observable. The code used for automated AI completion is made publicly available for further research.

Motivation & Objective

• Motivate and quantify the risk of AI-assisted exam completion in university Physics.
• Assess the performance of state-of-the-art LLMs (GPT-4 and GPT-3.5) on real Durham Physics exams from 2018–2022.
• Provide a transparent, reproducible methodology and open-source tooling for replication and further research.

Proposed method

• Automated extraction of individual questions from lecture-style LaTeX source files using regular expressions.
• Cleaning and LaTeX error correction with GPT-3.5 to ensure compilable inputs.
• Sending questions to OpenAI API with system prompts to assume a Physics professor role and produce LaTeX-formatted answers.
• Compiling AI outputs into per-exam PDFs and having them marked by the original course markers.
• Iterative LaTeX compilation checks with up to three retries; recording compilation failures and question-specific access issues.
• Manual verification of extracted questions and answers to ensure script reliability; sharing code on GitHub for replicability.

Experimental results

Research questions

• RQ1 Do GPT-4 and GPT-3.5 achieve non-trivial scores on Durham University Physics exams across multiple courses and levels?
• RQ2 How does AI performance differ between pre-COVID (in-person) and post-COVID (open-book/remote-adapted) exam formats?
• RQ3 Does AI performance vary with exam level (Levels 1–4) or by course type?
• RQ4 What factors correlate with higher or lower AI scores (e.g., presence of graphics, request to explain, mathematical language)?

Key findings

• GPT-4 averaged 49.4% and GPT-3.5 averaged 38.6% across 593 questions.
• Pre-COVID averages were 50.8% (GPT-4) and 41.6% (GPT-3.5).
• Post-COVID averages were 47.5% (GPT-4) and 33.6% (GPT-3.5).
• GPT-4 outperformed GPT-3.5 on all exam types, with closer results for Foundations of Physics 3A and Theoretical Astrophysics.
• Excluding zero-scores raises AI performance to 65.6% (GPT-4) and 56.7% (GPT-3.5) on non-zero attempts.
• The study provides open-source code for replication and emphasizes ongoing assessment of AI risks as models improve.