[论文解读] Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT

Chunqiu Steven Xia, Lingming Zhang|arXiv (Cornell University)|Apr 1, 2023

Software Testing and Debugging Techniques被引用 43

一句话总结

引入 ChatRepair，一种对话驱动的 APR 方法，在补丁生成与立即反馈之间穿插，使用 ChatGPT，实现 Defects4j 上的最新修复水平，并在337个 bugs 中修复162个，成本为每个 bug 0.42 美元。

ABSTRACT

Automated Program Repair (APR) aims to automatically generate patches for buggy programs. Recent APR work has been focused on leveraging modern Large Language Models (LLMs) to directly generate patches for APR. Such LLM-based APR tools work by first constructing an input prompt built using the original buggy code and then queries the LLM to generate patches. While the LLM-based APR tools are able to achieve state-of-the-art results, it still follows the classic Generate and Validate repair paradigm of first generating lots of patches and then validating each one afterwards. This not only leads to many repeated patches that are incorrect but also miss the crucial information in test failures as well as in plausible patches. To address these limitations, we propose ChatRepair, the first fully automated conversation-driven APR approach that interleaves patch generation with instant feedback to perform APR in a conversational style. ChatRepair first feeds the LLM with relevant test failure information to start with, and then learns from both failures and successes of earlier patching attempts of the same bug for more powerful APR. For earlier patches that failed to pass all tests, we combine the incorrect patches with their corresponding relevant test failure information to construct a new prompt for the LLM to generate the next patch. In this way, we can avoid making the same mistakes. For earlier patches that passed all the tests, we further ask the LLM to generate alternative variations of the original plausible patches. In this way, we can further build on and learn from earlier successes to generate more plausible patches to increase the chance of having correct patches. While our approach is general, we implement ChatRepair using state-of-the-art dialogue-based LLM -- ChatGPT. By calculating the cost of accessing ChatGPT, we can fix 162 out of 337 bugs for \$0.42 each!

研究动机与目标

• 激发并解决传统和现有基于大型语言模型的 APR 方法的局限性，这些方法依赖静态提示且缺乏测试失败集成。
• 提出一个完全自动化的对话驱动 APR 工作流，利用测试失败信息和过去的补丁历史来指导补丁生成。
• 展示将补丁生成与即时反馈交错进行相比先前方法能提高补丁质量与多样性。
• 通过计算每个修复 bug 的 ChatGPT API 使用量来展示成本感知评估。

提出的方法

• 提出 ChatRepair，一种基于 ChatGPT 的对话驱动 APR 工具，在带有反馈循环的对话中生成补丁。
• 提取简洁、相关的测试失败信息来为初始提示提供种子并指导补丁生成。
• 将补丁生成与基于测试的即时反馈交错进行，以避免重复失败的补丁并从错误中学习。
• 对于可行的补丁，生成替代表达以扩展候选修复集合。
• 在 Defects4j 1.2、2.0 和 QuixBugs 上进行评估，以建立处于领先地位的修复性能并计算每个 bug 的成本。
• 通过使用基于提示的交互来支持通用性，该交互能够合并丰富的测试失败信息和补丁历史。

Figure 1. Example bug fix with original testcase information

实验结果

研究问题

• RQ1 与传统的 Generate-and-Validate 方法相比，基于对话的与 LLM 的交互是否可以改善自动补丁生成？
• RQ2 将测试失败信息和先前的补丁历史纳入是否能够产生更合理和更正确的补丁？
• RQ3 在标准基准上使用基于对话的 APR 与 ChatGPT 时，修复性能提升和相关成本是什么？

主要发现

• ChatRepair 实现了最先进的修复性能，在 Defects4j 1.2 上有 114 个正确修复，在 Defects4j 2.0 上有 48 个。
• 该方法在 337 个 bug 中修复了 162 个，使用 ChatGPT API 访问的成本为每个 bug 0.42 美元。
• 结合丰富的测试失败信息和对话历史能够带来更好的补丁质量和更丰富的可行补丁。
• 消融研究显示使用失败信息和对话范式带来收益。
• 评估包括 Defects4j 1.2、2.0 和 QuixBugs 数据集，显示优于先前基线的性能。

Figure 2. Overview of ChatRepair