[Paper Review] Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code

Jaromír Šavelka, Arav Agarwal+2 more|arXiv (Cornell University)|Mar 9, 2023

Software Engineering Research4 citations

TL;DR

This study evaluates GPT models' performance on multiple-choice questions (MCQs) involving code snippets from introductory and intermediate Python courses. Despite strong performance on natural language MCQs (77.9% accuracy), GPT models struggle significantly with code-related reasoning tasks, achieving only 59.5% accuracy on code-containing MCQs, with the lowest performance on questions requiring execution trace or logical analysis (56.6%).

ABSTRACT

We analyzed effectiveness of three generative pre-trained transformer (GPT) models in answering multiple-choice question (MCQ) assessments, often involving short snippets of code, from introductory and intermediate programming courses at the postsecondary level. This emerging technology stirs countless discussions of its potential uses (e.g., exercise generation, code explanation) as well as misuses in programming education (e.g., cheating). However, the capabilities of GPT models and their limitations to reason about and/or analyze code in educational settings have been under-explored. We evaluated several OpenAI's GPT models on formative and summative MCQ assessments from three Python courses (530 questions). We found that MCQs containing code snippets are not answered as successfully as those that only contain natural language. While questions requiring to fill-in a blank in the code or completing a natural language statement about the snippet are handled rather successfully, MCQs that require analysis and/or reasoning about the code (e.g., what is true/false about the snippet, or what is its output) appear to be the most challenging. These findings can be leveraged by educators to adapt their instructional practices and assessments in programming courses, so that GPT becomes a valuable assistant for a learner as opposed to a source of confusion and/or potential hindrance in the learning process.

Motivation & Objective

• To investigate the effectiveness of GPT models in answering multiple-choice questions (MCQs) involving code snippets in programming education.
• To identify which types of MCQs are most challenging for GPT models, particularly those requiring reasoning about code behavior or execution.
• To compare the performance of different GPT variants (text-davinci-001, text-davinci-002, text-davinci-003) on code-related assessments.
• To provide empirical insights for educators on how to responsibly integrate GPT tools into programming instruction and assessment.
• To establish a foundation for understanding the limitations of LLMs in reasoning about program semantics, especially execution traces and logical properties.

Proposed method

• Collected 530 MCQs from three postsecondary Python courses, manually curated and categorized by question type.
• Classified questions into types such as true/false, output prediction, and fill-in-the-blank based on their cognitive demand.
• Evaluated three GPT models—text-davinci-001, text-davinci-002, and text-davinci-003—on the same MCQ set using zero-shot prompting.
• Measured performance using exact match and Jaccard similarity metrics, with a baseline of 23.4% Jaccard similarity for random guessing.
• Used pattern matching and manual validation to ensure consistent labeling and reliable performance measurement.
• Analyzed performance differences across question types to isolate cognitive challenges, such as execution simulation and intent inference.

Figure 1: The upper-left screenshot depicts a conversation with ChatGPT when asked to explain a code snippet line by line. It correctly explains the bahavior (1). The lower-right shows a conversation with ChatGPT when asked to generate the code snippet with the same behavior. The generated code is c

Experimental results

Research questions

• RQ1 Is there a significant difference in GPT model performance between MCQs containing code snippets and those with only natural language?
• RQ2 Which types of MCQs (e.g., output prediction, true/false, fill-in-the-blank) are most challenging for GPT models when reasoning about code?
• RQ3 How does the performance of newer GPT models (e.g., text-davinci-003) compare to older versions (e.g., text-davinci-001) on code-related reasoning tasks?
• RQ4 To what extent do GPT models fail to simulate code execution or reason about program behavior, and how does this affect their accuracy?

Key findings

• The most capable GPT model, text-davinci-003, achieved an overall accuracy of 65.5% on 530 MCQs, significantly outperforming the 23.4% Jaccard similarity baseline.
• MCQs containing code snippets were answered with 59.5% accuracy, substantially lower than the 77.9% accuracy on natural language-only MCQs.
• Fill-in-the-blank or sentence-completion questions were handled most successfully (87.1% accuracy), indicating strong pattern-matching and completion abilities.
• MCQs requiring reasoning about code behavior—such as predicting output or identifying true/false statements about execution—achieved only 56.6% accuracy, indicating core limitations in execution simulation.
• The performance gap between code and non-code MCQs suggests that GPT models struggle with mental simulation of control and data flow, even when they can explain or generate code correctly.
• Despite strong performance on code explanation and generation, GPT models fail on tasks requiring trace-based reasoning, highlighting a critical gap in their functional understanding of code.

Figure 2: Distribution of MCQs into categories. Note that the MCQs asking about the output of a code snippet as well as MCQs focused on filling-in the blanks in a snippet are not present in the MCQs with no code. This is to be expected given the nature of those questions. The MCQs with code are quit