[Paper Review] Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models

Tung Phung, José Cambronero+5 more|arXiv (Cornell University)|Jan 24, 2023

Software Engineering Research17 citations

TL;DR

The paper introduces PyFiXV, a Codex-powered system that generates high-precision feedback for Python syntax errors, with a run-time validation knob to control precision and coverage.

ABSTRACT

Large language models (LLMs), such as Codex, hold great promise in enhancing programming education by automatically generating feedback for students. We investigate using LLMs to generate feedback for fixing syntax errors in Python programs, a key scenario in introductory programming. More concretely, given a student's buggy program, our goal is to generate feedback comprising a fixed program along with a natural language explanation describing the errors/fixes, inspired by how a human tutor would give feedback. While using LLMs is promising, the critical challenge is to ensure high precision in the generated feedback, which is imperative before deploying such technology in classrooms. The main research question we study is: Can we develop LLMs-based feedback generation techniques with a tunable precision parameter, giving educators quality control over the feedback that students receive? To this end, we introduce PyFiXV, our technique to generate high-precision feedback powered by Codex. The key idea behind PyFiXV is to use a novel run-time validation mechanism to decide whether the generated feedback is suitable for sharing with the student; notably, this validation mechanism also provides a precision knob to educators. We perform an extensive evaluation using two real-world datasets of Python programs with syntax errors and show the efficacy of PyFiXV in generating high-precision feedback.

Motivation & Objective

• Motivate automated, precise feedback generation for syntax errors in introductory programming.
• Formalize feedback as a fixed program plus a natural language explanation.
• Develop PyFiXV with a run-time validation mechanism that acts as a precision knob for educators.
• Evaluate PyFiXV on real-world Python datasets and public release of the implementation.

Proposed method

• Stage-1 fixes buggy programs to a syntactically correct fixed program using Codex-Edit.
• Stage-2 generates natural language explanations using Codex-Complete with carefully chosen few-shot examples.
• Stage-3 validates the generated feedback by querying Codex-Edit to simulate a student model and decides acceptance based on AST-based exact matches.
• Introduce a precision-coverage trade-off via Tunable hyperparameters in Stage-3 to meet desired precision levels.
• Use a calibration dataset to select hyperparameters that achieve target precision while maximizing coverage.

Experimental results

Research questions

• RQ1 Can we develop LLM-based feedback generation techniques with a tunable precision parameter for high-quality feedback?
• RQ2 How does run-time validation impact the precision and coverage of feedback for syntax errors?
• RQ3 What is the effect of few-shot example selection on the quality of explanations generated by Codex-Complete?
• RQ4 How does PyFiXV perform on real-world Python datasets in terms of precision and coverage?

Key findings

• PyFiXV achieves high precision with a tunable knob that trades off coverage, demonstrated on TigerJython and Codeforces datasets.
• Stage-1 fixes a buggy program with small edits and high feasibility (high feedback generation success).
• Stage-2 explanations, aided by few-shot examples, improve the relevance and correctness of explanations.
• Stage-3 validation using a simulated student model (Codex-Edit) provides a robust precision mechanism, outperforming rule-based validation in experiments.
• Calibrated PyFiXV variants reach competitive precision-coverage trade-offs across datasets.
• Compared variants show that the proposed validation mechanism substantially improves precision over baselines like length-based validation.