[Paper Review] Automatic Repair of Real Bugs: An Experience Report on the Defects4J Dataset

Matías Martínez, Thomas Durieux+3 more|arXiv (Cornell University)|May 26, 2015

Software Testing and Debugging Techniques51 references47 citations

TL;DR

This paper evaluates state-of-the-art automatic repair tools—jGenProg, jKali, and Nopol—on the Defects4J dataset of 224 real Java bugs. It finds that 47 bugs can be automatically repaired, but only 9 of the 84 generated patches are manually verified as correct, revealing significant issues with test-suite-based repair due to under-specification and overfitting.

ABSTRACT

Defects4J is a large, peer-reviewed, structured dataset of real-world Java bugs. Each bug in Defects4J is provided with a test suite and at least one failing test case that triggers the bug. In this paper, we report on an experiment to explore the effectiveness of automatic repair on Defects4J. The result of our experiment shows that 47 bugs of the Defects4J dataset can be automatically repaired by state-of- the-art repair. This sets a baseline for future research on automatic repair for Java. We have manually analyzed 84 different patches to assess their real correctness. In total, 9 real Java bugs can be correctly fixed with test-suite based repair. This analysis shows that test-suite based repair suffers from under-specified bugs, for which trivial and incorrect patches still pass the test suite. With respect to practical applicability, it takes in average 14.8 minutes to find a patch. The experiment was done on a scientific grid, totaling 17.6 days of computation time. All their systems and experimental results are publicly available on Github in order to facilitate future research on automatic repair.

Motivation & Objective

• To assess the practical effectiveness of state-of-the-art automatic repair tools on real-world Java bugs from a large, peer-reviewed dataset.
• To investigate whether test-suite-based repair techniques generate patches that are not only test-passing but also semantically correct and meaningful.
• To identify bugs in Defects4J that are under-specified, where test suites fail to capture required behavior, leading to incorrect or trivial patches.
• To measure the computational cost of automatic repair on real industrial-scale codebases to evaluate practical applicability.

Proposed method

• Re-implemented three repair tools—jGenProg, jKali, and Nopol—based on GenProg and Kali approaches for Java.
• Executed the tools on 224 real Java bugs from the Defects4J dataset, each with a test suite and at least one failing test case.
• Used a scientific grid (Grid’5000) to run experiments over 17.6 days of total computation time.
• Manually analyzed 84 generated patches to assess correctness beyond test-suite passing, classifying them as correct, incorrect, or requiring expert knowledge.
• Applied statement ranking and patch validation techniques to guide search in the repair space.
• Collected and published all tools, configurations, and results on GitHub to ensure reproducibility and support future research.

Experimental results

Research questions

• RQ1 RQ1: Can state-of-the-art repair techniques automatically fix real bugs in the Defects4J dataset?
• RQ2 RQ2: Are the generated patches semantically correct, or do they overfit the test cases and fail to address the actual bug?
• RQ3 RQ3: Which bugs in Defects4J are under-specified, making them hard to repair using only test-suite feedback?
• RQ4 RQ4: What is the average execution time for repairing a single bug using these tools?

Key findings

• 47 out of 224 bugs in the Defects4J dataset were successfully repaired by at least one of the three tools, with Nopol fixing the most (35 bugs).
• Only 11 out of 84 generated patches were manually verified as correct, indicating that 72.6% of patches were incorrect or overly specific to test inputs.
• 12 of the 84 patches required expert knowledge for validation, highlighting the difficulty of automated correctness assessment.
• The average repair time per bug was 14.8 minutes, indicating that while feasible, the process is computationally intensive for industrial use.
• The study identified that 12 bugs were repaired by all three tools, suggesting a core set of reproducible fixes, while 35 were fixed only by Nopol.
• The results reveal a critical flaw in test-suite-based repair: weak test suites lead to incorrect patches that pass tests but do not semantically fix the bug.