[Paper Review] Exploring the Potential of Large Language Models to Generate Formative Programming Feedback

Natalie Kiesler, Dominic Lohr+1 more|arXiv (Cornell University)|Aug 31, 2023

Software Engineering Research4 citations

TL;DR

This study investigates the potential of large language models (LLMs), specifically ChatGPT, to generate formative programming feedback for introductory programming students. Using real student code sequences from a CS1 course, the authors evaluate how effectively ChatGPT provides feedback and correct solutions, finding it reasonably effective for some tasks but cautioning that misleading information may mislead novice learners without proper guidance.

ABSTRACT

Ever since the emergence of large language models (LLMs) and related applications, such as ChatGPT, its performance and error analysis for programming tasks have been subject to research. In this work-in-progress paper, we explore the potential of such LLMs for computing educators and learners, as we analyze the feedback it generates to a given input containing program code. In particular, we aim at (1) exploring how an LLM like ChatGPT responds to students seeking help with their introductory programming tasks, and (2) identifying feedback types in its responses. To achieve these goals, we used students' programming sequences from a dataset gathered within a CS1 course as input for ChatGPT along with questions required to elicit feedback and correct solutions. The results show that ChatGPT performs reasonably well for some of the introductory programming tasks and student errors, which means that students can potentially benefit. However, educators should provide guidance on how to use the provided feedback, as it can contain misleading information for novices.

Motivation & Objective

• To examine how large language models like ChatGPT respond to students seeking help with introductory programming tasks.
• To identify and categorize the types of feedback generated by LLMs in response to student code.
• To assess the reliability and educational value of LLM-generated feedback for novice programmers.
• To highlight the risks of misinformation in LLM feedback and the need for educator guidance.

Proposed method

• The study used student programming sequences collected from a CS1 course as input for ChatGPT.
• Feedback and correct solutions were elicited by posing specific questions alongside the code submissions.
• The LLM responses were analyzed to identify feedback types and assess accuracy and helpfulness.
• The analysis focused on common introductory programming errors and task types.
• The researchers evaluated the quality of feedback based on correctness, clarity, and pedagogical relevance.

Experimental results

Research questions

• RQ1 How does ChatGPT respond to student code submissions in introductory programming tasks?
• RQ2 What types of feedback does ChatGPT generate when asked to correct student code?
• RQ3 How accurate and helpful is the feedback provided by ChatGPT for novice programming errors?
• RQ4 In what ways might LLM-generated feedback mislead novice learners?

Key findings

• ChatGPT generated reasonably accurate feedback for a subset of introductory programming tasks and common student errors.
• The model produced feedback that was helpful in some cases, particularly for syntax and logic errors in basic programming constructs.
• However, the model occasionally provided incorrect or misleading feedback, especially for more complex or ambiguous code.
• Feedback quality varied significantly depending on the clarity of the input and the nature of the programming error.
• The study highlights the risk of novice learners accepting incorrect feedback without critical evaluation.