[Paper Review] Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT

Chunqiu Steven Xia, Lingming Zhang|arXiv (Cornell University)|Apr 1, 2023

Software Testing and Debugging Techniques43 citations

TL;DR

Introduces ChatRepair, a conversation-driven APR approach that interleaves patch generation with immediate feedback using ChatGPT, achieving state-of-the-art fixes on Defects4j and fixing 162 of 337 bugs at a cost of $0.42 per bug.

ABSTRACT

Automated Program Repair (APR) aims to automatically generate patches for buggy programs. Recent APR work has been focused on leveraging modern Large Language Models (LLMs) to directly generate patches for APR. Such LLM-based APR tools work by first constructing an input prompt built using the original buggy code and then queries the LLM to generate patches. While the LLM-based APR tools are able to achieve state-of-the-art results, it still follows the classic Generate and Validate repair paradigm of first generating lots of patches and then validating each one afterwards. This not only leads to many repeated patches that are incorrect but also miss the crucial information in test failures as well as in plausible patches. To address these limitations, we propose ChatRepair, the first fully automated conversation-driven APR approach that interleaves patch generation with instant feedback to perform APR in a conversational style. ChatRepair first feeds the LLM with relevant test failure information to start with, and then learns from both failures and successes of earlier patching attempts of the same bug for more powerful APR. For earlier patches that failed to pass all tests, we combine the incorrect patches with their corresponding relevant test failure information to construct a new prompt for the LLM to generate the next patch. In this way, we can avoid making the same mistakes. For earlier patches that passed all the tests, we further ask the LLM to generate alternative variations of the original plausible patches. In this way, we can further build on and learn from earlier successes to generate more plausible patches to increase the chance of having correct patches. While our approach is general, we implement ChatRepair using state-of-the-art dialogue-based LLM -- ChatGPT. By calculating the cost of accessing ChatGPT, we can fix 162 out of 337 bugs for \$0.42 each!

Motivation & Objective

• Motivate and address limitations of traditional and existing LLM-based APR approaches that rely on static prompts and lack test-failure integration.
• Propose a fully automated conversation-driven APR workflow that leverages test failure information and past patch history to guide patch generation.
• Show that interleaving patch generation with instant feedback improves patch quality and diversity over prior methods.
• Demonstrate cost-aware evaluation by calculating ChatGPT API usage per fixed bug.

Proposed method

• Propose ChatRepair, a conversation-driven APR tool built on ChatGPT to generate patches in a dialog with feedback loops.
• Extract concise, relevant test failure information to seed the initial prompt and guide patch generation.
• Interleave patch generation with immediate test-based feedback to avoid repeating failed patches and to learn from errors.
• For plausible patches, generate alternative variations to expand the set of candidate fixes.
• Evaluate on Defects4j 1.2, 2.0 and QuixBugs to establish state-of-the-art repair performance and compute per-bug cost.
• Support generality by using prompting-based interaction that can incorporate rich test failure information and patch history.

Figure 1. Example bug fix with original testcase information

Experimental results

Research questions

• RQ1 Can a conversation-driven interaction with an LLM improve automatic patch generation compared to traditional Generate-and-Validate approaches?
• RQ2 Does incorporating test failure information and prior patch history enable more plausible and correct patches?
• RQ3 What are the repair performance gains and the associated costs when using a dialogue-based APR with ChatGPT on standard benchmarks?

Key findings

• ChatRepair achieves state-of-the-art repair performance, with 114 correct fixes on Defects4j 1.2 and 48 on Defects4j 2.0.
• The approach fixes 162 out of 337 bugs at a cost of $0.42 per bug using ChatGPT API access.
• Incorporating rich test failure information and conversational history leads to better patch quality and more diverse plausible patches.
• Ablation studies show gains from using failure information and the conversational paradigm.
• Evaluations include Defects4j 1.2, 2.0 and QuixBugs datasets, demonstrating superior performance over prior baselines.

Figure 2. Overview of ChatRepair