[Paper Review] Comparing Code Explanations Created by Students and Large Language Models

Juho Leinonen, Paul Denny+6 more|arXiv (Cornell University)|Apr 8, 2023

Software Engineering Research40 references10 citations

TL;DR

The study compares student-created code explanations with GPT-3 generated explanations in a large CS1 course and finds LLM explanations are rated as more accurate and easier to understand, with similar length.

ABSTRACT

Reasoning about code and explaining its purpose are fundamental skills for computer scientists. There has been extensive research in the field of computing education on the relationship between a student's ability to explain code and other skills such as writing and tracing code. In particular, the ability to describe at a high-level of abstraction how code will behave over all possible inputs correlates strongly with code writing skills. However, developing the expertise to comprehend and explain code accurately and succinctly is a challenge for many students. Existing pedagogical approaches that scaffold the ability to explain code, such as producing exemplar code explanations on demand, do not currently scale well to large classrooms. The recent emergence of powerful large language models (LLMs) may offer a solution. In this paper, we explore the potential of LLMs in generating explanations that can serve as examples to scaffold students' ability to understand and explain code. To evaluate LLM-created explanations, we compare them with explanations created by students in a large course ($n \approx 1000$) with respect to accuracy, understandability and length. We find that LLM-created explanations, which can be produced automatically on demand, are rated as being significantly easier to understand and more accurate summaries of code than student-created explanations. We discuss the significance of this finding, and suggest how such models can be incorporated into introductory programming education.

Motivation & Objective

• Motivate the need to scale code explanation scaffolding in large CS classrooms.
• Investigate whether LLM-generated code explanations can match or exceed student-created explanations in accuracy and understandability.
• Examine which aspects of code explanations students find most useful and how explanations are valued.
• Assess whether LLM explanations can serve as scalable exemplars for novices learning to explain code.

Proposed method

• Use a large first-year course (~1000 students) to collect explanations of three functions.
• Have students create explanations for three functions (Lab A) and then rate a random sample of 54 explanations (Lab B) from both students and GPT-3.
• Evaluate explanations via three 5-point Likert questions: ease of understanding, accuracy of summary, and ideal length.
• Compare lengths in characters to establish baseline differences.
• Apply Mann–Whitney U tests with Bonferroni correction to assess differences between sources.
• Conduct thematic analysis on open-ended responses from students to identify valued qualities in explanations.

Experimental results

Research questions

• RQ1 RQ1 To what extent do code explanations created by students and LLMs differ in accuracy, length, and understandability?
• RQ2 RQ2 What aspects of code explanations do students value?

Key findings

• LLM-generated explanations are rated as more accurate than student-created ones.
• LLM-generated explanations are rated as easier to understand than student-created ones.
• There is no statistically significant difference in perceived or actual length between student- and LLM-generated explanations.
• The ideal length ratings do not differ significantly between sources after correction.
• In open responses, students prefer line-by-line explanations and value explanations that state inputs/outputs and describe the code’s purpose.