[Paper Review] Neural Program Repair by Jointly Learning to Localize and Repair

Marko Vasić, Aditya Kanade+3 more|arXiv (Cornell University)|Apr 3, 2019

Advanced Memory and Neural Computing44 citations

TL;DR

The paper introduces a joint neural model that localizes and repairs VarMisuse bugs with multi-headed pointer networks, outperforming an enumerative repair approach and a graph-based baseline.

ABSTRACT

Due to its potential to improve programmer productivity and software quality, automated program repair has been an active topic of research. Newer techniques harness neural networks to learn directly from examples of buggy programs and their fixes. In this work, we consider a recently identified class of bugs called variable-misuse bugs. The state-of-the-art solution for variable misuse enumerates potential fixes for all possible bug locations in a program, before selecting the best prediction. We show that it is beneficial to train a model that jointly and directly localizes and repairs variable-misuse bugs. We present multi-headed pointer networks for this purpose, with one head each for localization and repair. The experimental results show that the joint model significantly outperforms an enumerative solution that uses a pointer based model for repair alone.

Motivation & Objective

• Motivate automated repair of variable-misuse bugs (VarMisuse) to improve programmer productivity and software quality.
• Replace enumerative repair with a model that jointly classifies, localizes, and repairs bugs.
• Leverage pointer networks to point to bug locations and repair variables within the input program.
• Evaluate the proposed joint model against prior enumerative approaches and graph-based baselines on large Python and C# datasets.

Proposed method

• Use multi-headed pointer networks on top of an LSTM encoder to jointly predict bug location and repair variable.
• Represent programs as token sequences with embeddings for tokens and variables.
• Train with synthetic buggy and non-buggy examples derived from correct programs, using Loc and Rep pointers as supervision.
• Define VarMisuseRepair to output a location token (or no-fault) and a repair token from within the program scope.
• Compare joint model to an enumerative repair approach that predicts repairs per slot and aggregates results.
• Evaluate using ETH-Py150 (Python) and MSR-VarMisuse (C#) datasets, with metrics for true positives, localization, and localization+repair accuracy.

Experimental results

Research questions

• RQ1 Can a joint VarMisuseRepair model simultaneously localize and repair bugs more effectively than an enumerative repair approach?
• RQ2 How does the presence of unknown bugs affect repair accuracy in non-enumerative settings?
• RQ3 How does a pointer-network based approach compare to a graph-based model for VarMisuse repair when limited to syntactic information?
• RQ4 What is the impact of incorrect slot placement on repair performance in enumerative versus joint models.

Key findings

• The joint model achieves higher localization accuracy (71%) and localization+repair accuracy (65.7%) than the enumerative baseline under various settings, with true-positive and classification accuracy of 84.5% and 82.4%, respectively.
• Enumerative approaches suffer notable drops in localization and repair accuracy when predicted slots are incorrect or when thresholds are adjusted, e.g., localization drops from 64.6% to 7.0% under certain thresholds.
• On ETH-Py150, the joint model improves localization by about 6.4 percentage points and localization+repair by about 9.9 points over the lowest-threshold/highest-top-k configuration of the enumerative method.
• Pointer-network based joint modeling outperforms a graph-based repair model when restricted to syntactic inputs (62.3% vs 55.3% seen test on MSR-VarMisuse in their ablation).
• The model learns to classify, localize, and repair without explicit enumeration, offering efficiency advantages during training and inference.