[Paper Review] Explaining Code with a Purpose: An Integrated Approach for Developing Code Comprehension and Prompting Skills

Paul Denny, David H. Smith+4 more|arXiv (Cornell University)|Mar 10, 2024

Software Engineering Research4 citations

TL;DR

This paper proposes an integrated pedagogical approach that uses LLMs to automatically assess students' 'Explain in Plain English' (EiPE) responses by generating code from their explanations, thereby evaluating code comprehension and prompting skills in parallel. In a large CS1 course, students achieved high success rates in crafting effective prompts that produced functionally equivalent code, with relational, purpose-focused explanations proving most effective.

ABSTRACT

Reading, understanding and explaining code have traditionally been important skills for novices learning programming. As large language models (LLMs) become prevalent, these foundational skills are more important than ever given the increasing need to understand and evaluate model-generated code. Brand new skills are also needed, such as the ability to formulate clear prompts that can elicit intended code from an LLM. Thus, there is great interest in integrating pedagogical approaches for the development of both traditional coding competencies and the novel skills required to interact with LLMs. One effective way to develop and assess code comprehension ability is with ``Explain in plain English'' (EiPE) questions, where students succinctly explain the purpose of a fragment of code. However, grading EiPE questions has always been difficult given the subjective nature of evaluating written explanations and this has stifled their uptake. In this paper, we explore a natural synergy between EiPE questions and code-generating LLMs to overcome this limitation. We propose using an LLM to generate code based on students' responses to EiPE questions -- not only enabling EiPE responses to be assessed automatically, but helping students develop essential code comprehension and prompt crafting skills in parallel. We investigate this idea in an introductory programming course and report student success in creating effective prompts for solving EiPE questions. We also examine student perceptions of this activity and how it influences their views on the use of LLMs for aiding and assessing learning.

Motivation & Objective

• To address the longstanding challenge of subjective and time-consuming grading of EiPE (Explain in Plain English) questions in programming education.
• To develop a scalable, automated method for assessing code comprehension using LLMs that generate code from student explanations.
• To simultaneously foster students' ability to craft effective prompts for LLMs, a critical emerging skill in modern programming.
• To investigate whether EiPE-based LLM prompting tasks can enhance student engagement and understanding of code beyond traditional coding exercises.
• To explore the potential of this method for supporting diverse learners, including those with non-native English proficiency, through multilingual explanation support.

Proposed method

• Students are presented with a code fragment and asked to explain its purpose in plain English, emulating traditional EiPE questions.
• Student explanations are fed as prompts to a code-generating LLM to produce corresponding code.
• Generated code is automatically evaluated for functional equivalence to the original using a test suite, enabling objective assessment.
• The method leverages the SOLO taxonomy to classify the quality of student explanations based on their integration of code elements.
• Success is measured by whether the LLM-generated code passes all test cases, indicating accurate comprehension.
• The approach supports formative assessment and skill development in both code comprehension and prompt engineering.

Experimental results

Research questions

• RQ1 To what extent do students successfully generate prompts that result in functionally equivalent code when explaining code fragments in plain English?
• RQ2 How does the quality of student explanations—measured by SOLO taxonomy levels—affect the success rate of LLM-generated code?
• RQ3 What are students' perceptions of this LLM-assisted EiPE task compared to traditional programming exercises?
• RQ4 Can this method support equitable assessment for students with varying levels of English proficiency through multilingual explanations?
• RQ5 How do prompt length and structure influence the reliability and correctness of LLM-generated code?

Key findings

• In a large CS1 course (n ≈ 900), students achieved high success rates in generating prompts that produced functionally correct code from EiPE explanations.
• Higher-level, relational explanations—those integrating multiple code elements meaningfully—were significantly more likely to result in correct code generation than surface-level or line-by-line descriptions.
• Students reported that the activity was engaging and effective for improving code comprehension, and many found it novel and valuable for learning.
• The method enabled automatic, objective assessment of EiPE responses, overcoming the long-standing barrier of subjective grading.
• Six successful student explanations were submitted in Chinese, demonstrating the potential of the approach to support multilingual learners and reduce language-based assessment bias.
• Despite high success rates, prompt reliability varied, with the same prompt sometimes generating correct or incorrect code, indicating a need for improved consistency checks.