[Paper Review] AI-enhanced Auto-correction of Programming Exercises: How Effective is GPT-3.5?

Imen Azaiz, Oliver Deckarm+1 more|arXiv (Cornell University)|Oct 24, 2023

Artificial Intelligence in Healthcare and Education26 references4 citations

TL;DR

This study evaluates GPT-3.5's effectiveness in generating personalized feedback for introductory programming assignments by analyzing real student submissions. It finds that GPT-3.5 correctly identifies correctness in 73% of cases and provides high-quality feedback in 59% of those, though it occasionally misidentifies errors or suggests changes violating assignment instructions.

ABSTRACT

Timely formative feedback is considered as one of the most important drivers for effective learning. Delivering timely and individualized feedback is particularly challenging in large classes in higher education. Recently Large Language Models such as GPT-3 became available to the public that showed promising results on various tasks such as code generation and code explanation. This paper investigates the potential of AI in providing personalized code correction and generating feedback. Based on existing student submissions of two different real-world assignments, the correctness of the AI-aided e-assessment as well as the characteristics such as fault localization, correctness of hints, and code style suggestions of the generated feedback are investigated. The results show that 73 % of the submissions were correctly identified as either correct or incorrect. In 59 % of these cases, GPT-3.5 also successfully generated effective and high-quality feedback. Additionally, GPT-3.5 exhibited weaknesses in its evaluation, including localization of errors that were not the actual errors, or even hallucinated errors. Implications and potential new usage scenarios are discussed.

Motivation & Objective

• To investigate whether GPT-3.5 can effectively deliver personalized, high-quality feedback on programming exercise submissions in large-scale higher education settings.
• To assess the accuracy of GPT-3.5 in determining whether student code is functionally correct or incorrect.
• To evaluate the quality and characteristics of feedback generated by GPT-3.5, including error localization, hint quality, code style suggestions, and alignment with assignment instructions.
• To identify strengths and weaknesses of GPT-3.5 in providing automated feedback, particularly in comparison to traditional e-assessment systems.
• To explore practical implications and limitations for integrating LLMs into programming education workflows, especially for formative assessment.

Proposed method

• Collected real student submissions from two first-semester programming assignments at LMU Munich.
• Submitted each submission along with assignment instructions to GPT-3.5 via a prompt-based interface using a consistent prompt structure.
• Conducted three independent GPT-3.5 inference runs per submission to mitigate response variability and aggregated results.
• Manually classified GPT-3.5 responses using a coding scheme developed by two researchers through iterative discussion to ensure reliability.
• Evaluated feedback quality based on correctness, error localization, hint relevance, compliance with instructions, and personalization.
• Compared GPT-3.5 outputs against ground-truth correctness labels and assignment requirements to measure accuracy and fidelity.

Experimental results

Research questions

• RQ1 RQ1: What is the quality of GPT-3.5 in determining the correctness of student submissions?
• RQ2 RQ2: How can the GPT-3.5-generated feedback be characterized in terms of length, hint quality, compliance with assignment instructions, error localization, code correction, and personalization?
• RQ3 RQ3: What are the strengths and weaknesses of GPT-3.5 in delivering personalized feedback and code corrections for programming exercises?

Key findings

• GPT-3.5 correctly identified the correctness of student submissions in 73% of cases, with higher accuracy in detecting incorrect submissions than correct ones.
• In 59% of cases where correctness was correctly assessed, GPT-3.5 generated feedback that was effective and of high quality, including accurate error localization and actionable suggestions.
• GPT-3.5 occasionally mislocalized errors, suggesting fixes for non-existent issues or failing to detect actual bugs, particularly in subtle aspects like string casing.
• The model sometimes provided feedback that violated assignment instructions, such as suggesting changes that altered the required behavior or output format.
• GPT-3.5 struggled with syntax errors in otherwise correct algorithms, sometimes overlooking them while focusing on functional logic.
• Despite limitations, GPT-3.5 demonstrated potential as a pre-assessment tool for teaching assistants, enabling rapid error identification and feedback draft generation.