[Paper Review] An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation

Michele Tufano, Cody Watson+4 more|arXiv (Cornell University)|Dec 20, 2018

Software Engineering Research4 citations

TL;DR

This paper proposes a neural machine translation (NMT) approach to automatically learn and generate real-world bug-fixing patches from millions of open-source code changes on GitHub. By training sequence-to-sequence models on abstracted buggy and fixed code pairs, the method achieves 9–50% fix prediction accuracy depending on candidate generation count, generates syntactically valid patches in over 82% of cases, and produces patches in milliseconds, demonstrating strong feasibility for scalable, data-driven program repair.

ABSTRACT

Millions of open-source projects with numerous bug fixes are available in code repositories. This proliferation of software development histories can be leveraged to learn how to fix common programming bugs. To explore such a potential, we perform an empirical study to assess the feasibility of using Neural Machine Translation techniques for learning bug-fixing patches for real defects. First, we mine millions of bug-fixes from the change histories of projects hosted on GitHub, in order to extract meaningful examples of such bug-fixes. Next, we abstract the buggy and corresponding fixed code, and use them to train an Encoder-Decoder model able to translate buggy code into its fixed version. In our empirical investigation we found that such a model is able to fix thousands of unique buggy methods in the wild. Overall, this model is capable of predicting fixed patches generated by developers in 9-50% of the cases, depending on the number of candidate patches we allow it to generate. Also, the model is able to emulate a variety of different Abstract Syntax Tree operations and generate candidate patches in a split second.

Motivation & Objective

• To investigate whether neural machine translation (NMT) can learn to generate realistic, developer-written bug-fixing patches from real-world code repositories.
• To extract and abstract meaningful buggy-fixed code pairs (BFPs) from the change histories of open-source projects on GitHub.
• To evaluate the feasibility, quality, and efficiency of NMT-based models in generating syntactically correct and semantically relevant patches for real defects.
• To assess the model's ability to emulate common Abstract Syntax Tree (AST) operations used in actual developer fixes.
• To explore the potential of leveraging large-scale, real-world code history for training data-driven program repair systems.

Proposed method

• The authors mine millions of bug-fix commits from GitHub repositories to collect real-world buggy and fixed code pairs.
• They abstract the source code by replacing identifiers and literals with placeholders, preserving syntactic and structural patterns while removing noise.
• An encoder-decoder neural network architecture is trained to translate buggy code (source) into fixed code (target), using sequence-to-sequence learning with attention mechanisms.
• The model is trained on 10% to 90% of the training data to assess scalability and performance trade-offs, though full training curves are left for future work.
• Model inference generates multiple candidate patches, and the best-performing patch is selected based on similarity to human-written fixes.
• The approach is evaluated on two datasets: small BFPs (≤100 tokens) and medium BFPs (101–200 tokens), with qualitative and quantitative analysis of patch correctness and AST operation coverage.

Experimental results

Research questions

• RQ1 Can neural machine translation effectively learn to generate real-world bug-fixing patches from the wild, based on actual developer changes in open-source repositories?
• RQ2 How accurate is the NMT model in predicting human-written patches, and how does performance vary with the number of candidate patches generated?
• RQ3 To what extent can the model emulate common Abstract Syntax Tree (AST) operations used in real bug fixes?
• RQ4 How efficient is the model in generating candidate patches, and can it scale to real-time use in development workflows?
• RQ5 What is the impact of training data size on model performance, and are there diminishing returns beyond certain data thresholds?

Key findings

• The NMT model successfully fixes 2,927 unique small BFPs (up to 100 tokens), achieving 9–50% prediction accuracy depending on the number of candidate patches generated.
• For medium BFPs (101–200 tokens), the model fixes 1,869 unique bugs with 3–28% accuracy under the same candidate generation conditions.
• Over 82% of the generated patches are syntactically valid, indicating strong structural correctness.
• Model $M_{small}$ emulates 28–64% of the AST operations performed in real fixes, while $M_{medium}$ achieves 16–52% coverage.
• The model generates tens of candidate patches in less than a second, demonstrating high inference efficiency suitable for real-time integration.
• The study establishes a strong empirical foundation for using NMT-based models to learn from real-world code change histories, enabling scalable and data-driven program repair.