[Paper Review] Conversational Automated Program Repair

Chunqiu Steven Xia, Lingming Zhang|arXiv (Cornell University)|Jan 30, 2023

Software Testing and Debugging TechniquesComputer Science32 citations

TL;DR

The paper introduces conversational APR, an LLM-based approach that alternates patch generation and validation with feedback, improving repair effectiveness over traditional sampling methods across multiple models and benchmarks.

ABSTRACT

Automated Program Repair (APR) can help developers automatically generate patches for bugs. Due to the impressive performance obtained using Large Pre-Trained Language Models (LLMs) on many code related tasks, researchers have started to directly use LLMs for APR. However, prior approaches simply repeatedly sample the LLM given the same constructed input/prompt created from the original buggy code, which not only leads to generating the same incorrect patches repeatedly but also miss the critical information in testcases. To address these limitations, we propose conversational APR, a new paradigm for program repair that alternates between patch generation and validation in a conversational manner. In conversational APR, we iteratively build the input to the model by combining previously generated patches with validation feedback. As such, we leverage the long-term context window of LLMs to not only avoid generating previously incorrect patches but also incorporate validation feedback to help the model understand the semantic meaning of the program under test. We evaluate 10 different LLM including the newly developed ChatGPT model to demonstrate the improvement of conversational APR over the prior LLM for APR approach.

Motivation & Objective

• Motivate reducing manual effort in automated program repair (APR) by leveraging LLMs with feedback from test results.
• Propose conversational APR that interleaves patch generation and validation to utilize long-context capabilities of LLMs.
• Show that incorporating validation feedback and previously generated patches improves patch diversity and success rate across multiple models.
• Evaluate conversational APR on a standard bug fix benchmark across 10 LLMs including ChatGPT to assess effectiveness and efficiency.

Proposed method

• Define a conversational turn as generating a patch, validating it, and extracting feedback.
• Construct input prompts that concatenate prior patches and their validation feedback to guide subsequent generations.
• Use a maximum chain length to manage LLM context and automatically restart with a fresh chain when needed.
• Compare conversational APR against a baseline sampling approach under identical prompts.
• Evaluate with multiple LLMs (CodeGen variants, Codex, ChatGPT) on QuixBugs for Python and Java.

Figure 1: Overview of conversational APR with an illustrative example in fixing the buggy sieve function

Experimental results

Research questions

• RQ1 RQ1: How effective is conversational APR compared with baseline sampling-based LLM APR?
• RQ2 RQ2: How do different components (chain length, validation prompts) affect performance?
• RQ3 RQ3: How does ChatGPT compare with other LLMs for conversational APR on the same benchmarks?
• RQ4 RQ4: What is the impact of prompt style and validation feedback on repair outcomes?

Key findings

• Conversational APR improves the number of correct and plausible patches across unsupervised LLMs of various sizes compared with baseline sampling.
• Longer chain length helps up to a model-dependent point (around 3–4 turns for CodeGen; larger models cope better with longer chains).
• Functional validation prompts (calling the function with test inputs) yield the best results among prompting styles.
• ChatGPT generally outperforms Codex in this setup, benefiting from its dialogue-focused training and RLHF.
• Using validation feedback reduces repeated patches and lowers the number of samples (tries) needed to reach plausible/correct patches.

Figure 2: Example bug in both Python and Java in QuixBugs along with the testcases