[Paper Review] An Analysis of the Automatic Bug Fixing Performance of ChatGPT

Dominik Sobania, Martin Briesch+2 more|arXiv (Cornell University)|Jan 20, 2023

Software Testing and Debugging Techniques49 citations

TL;DR

ChatGPT's bug-fixing performance on QuixBugs is competitive with Codex and CoCoNut and far better than standard APR; using dialogue hints markedly increases success (up to 31 of 40).

ABSTRACT

To support software developers in finding and fixing software bugs, several automated program repair techniques have been introduced. Given a test suite, standard methods usually either synthesize a repair, or navigate a search space of software edits to find test-suite passing variants. Recent program repair methods are based on deep learning approaches. One of these novel methods, which is not primarily intended for automated program repair, but is still suitable for it, is ChatGPT. The bug fixing performance of ChatGPT, however, is so far unclear. Therefore, in this paper we evaluate ChatGPT on the standard bug fixing benchmark set, QuixBugs, and compare the performance with the results of several other approaches reported in the literature. We find that ChatGPT's bug fixing performance is competitive to the common deep learning approaches CoCoNut and Codex and notably better than the results reported for the standard program repair approaches. In contrast to previous approaches, ChatGPT offers a dialogue system through which further information, e.g., the expected output for a certain input or an observed error message, can be entered. By providing such hints to ChatGPT, its success rate can be further increased, fixing 31 out of 40 bugs, outperforming state-of-the-art.

Motivation & Objective

• Evaluate ChatGPT's automatic bug-fixing performance on a standard benchmark (QuixBugs).
• Compare ChatGPT against Codex and traditional deep-learning APR methods (CoCoNut, Codex).
• Investigate how dialogue-driven hints affect ChatGPT's bug-fixing success.
• Characterize types of ChatGPT responses and the impact of follow-up interaction on repair rates.

Proposed method

• Use QuixBugs Python bugs (40 problems) as the benchmark.
• Query ChatGPT four times per problem with the prompt: 'Does this program have a bug? How to fix it?' without comments.
• Compare results to prior literature: standard APR methods from Ye et al. (2021) and DL-based methods CoCoNut (Lutellier et al.) and Codex (Prenner et al.).
• Classify ChatGPT answers into predefined categories and analyze response patterns (e.g., more information required, no bug found, correct fix).
• Conduct a dialogue study by providing hints to ChatGPT and measuring impact on repair rate (follow-up requests).

Experimental results

Research questions

• RQ1 How does ChatGPT perform on automatic bug fixing relative to Codex, CoCoNut, and standard APR approaches on QuixBugs?
• RQ2 Does incorporating a dialogue-based hint improve ChatGPT's bug-fixing success rate?
• RQ3 What types of responses does ChatGPT produce when asked to fix bugs, and how do these affect repair outcomes?
• RQ4 To what extent can follow-up information (hints) increase ChatGPT’s ability to fix bugs?

Key findings

• ChatGPT fixes 19 out of 40 bugs, comparable to Codex (21) and CoCoNut (19), and substantially better than standard APR (7).
• ChatGPT’s performance can improve with dialogue: providing hints increases fixes to 31 out of 40 across problems.
• ChatGPT’s results show high per-problem variance (often fixes appear in 1–2 runs; some problems fixed in all runs like bucketsort/flatten).
• Compared with traditional APR, DL-based models show higher bug-fix rates, likely due to broader search and learned repair patterns, while standard APR overfits to test suites.
• A dialogue-enabled interaction allows targeted information exchange (e.g., failing inputs or expected outputs) to greatly boost repair success.