[Paper Review] Self-Supervised Bug Detection and Repair

Miltiadis Allamanis, Henry Jackson-Flux+1 more|arXiv (Cornell University)|May 26, 2021

Software Engineering ResearchComputer Science33 references38 citations

TL;DR

BugLab jointly trains a bug selector to generate hard-to-detect bugs and a bug detector to locate and repair them, achieving better performance than random-bug baselines on real-world PyPI bugs and finding 19 previously unknown bugs in open-source Python code.

ABSTRACT

Machine learning-based program analyses have recently shown the promise of integrating formal and probabilistic reasoning towards aiding software development. However, in the absence of large annotated corpora, training these analyses is challenging. Towards addressing this, we present BugLab, an approach for self-supervised learning of bug detection and repair. BugLab co-trains two models: (1) a detector model that learns to detect and repair bugs in code, (2) a selector model that learns to create buggy code for the detector to use as training data. A Python implementation of BugLab improves by up to 30% upon baseline methods on a test dataset of 2374 real-life bugs and finds 19 previously unknown bugs in open-source software.

Motivation & Objective

• Address the lack of large annotated bug datasets by proposing a self-supervised framework for bug detection and repair.
• Co-train a bug selector that generates challenging bugs and a detector that repairs them.
• Demonstrate effectiveness on Python with a real-bug test set and open-source projects.
• Provide a Python implementation (PyBugLab) and release datasets for reproducibility.

Proposed method

• Represent code as a heterogeneous graph of entities and relations.
• Use two neural models: a selector S_phi to choose rewrites that introduce bugs and a detector D_theta to locate and repair them.
• Formulate a min-max training objective where the selector approximates the hardest rewrites for the detector (max loss) and the detector learns to repair (minimize loss).
• Employ probabilistic rewrites at locations ell with rewrite rules rho, including both bug-inducing and repair rewrites, with inverse rules rho^{-1} for repairing.
• Adopt two code representation architectures (GNN and GREAT transformer) to produce entity embeddings for rewrite scoring.
• Provide a Python implementation (PyBugLab) that handles four bug types: variable misuse, argument swapping, wrong operator, and wrong literal, plus data-augmentation rewrites.

Experimental results

Research questions

• RQ1 Can self-supervised co-training of a bug selector and a bug detector improve detection and repair performance compared to supervised or random-bug baselines?
• RQ2 To what extent can learned rewrites generate harder-to-detect bugs while maintaining realistic bug distributions?
• RQ3 How do different code representations (GNN vs GREAT) impact localization and repair performance on synthetic and real bug sets?
• RQ4 What is the relative difficulty of bug localization versus repair in real-world bug datasets?
• RQ5 How does PyBugLab perform on real-world PyPI bugs compared with existing approaches like CuBERT?

Key findings

• BugLab-trained detectors outperform baselines trained on randomly inserted bugs on a synthetic RandomBugs dataset.
• On RandomBugs, PyBugLab (GNN) achieves Joint=69.6%, Loc=80.4%, Repair=84.2%; PyBugLab + Aug yields Joint=70.3%, Loc=81.1%, Repair=84.5%.
• On the real Bug dataset PyPIBugs, PyBugLab (GNN) achieves Joint=24.2%, Loc=31.3%, Repair=70.7%; PyBugLab + Aug yields Joint=26.4%, Loc=33.5%, Repair=72.0%.
• Ensembling five GNN models improves localization to 83.0% and repair to 85.4% on RandomBugs, and increases Loc/Repair on PyPIBugs modestly.
• PyBugLab generally outperforms CuBERT in recall for bug localization, with higher overall detection capabilities across bug types.
• Further analysis shows that detecting NoBug is particularly challenging, with accuracy not exceeding ~63% on the PyPIBugs test set.
• The selector learns to generate harder-to-diagnose bugs over time, indicating its effectiveness in driving detector robustness.