[Paper Review] Break-It-Fix-It: Unsupervised Learning for Program Repair

Michihiro Yasunaga, Percy Liang|arXiv (Cornell University)|Jun 11, 2021

Domain Adaptation and Few-Shot Learning78 references28 citations

TL;DR

Break-It-Fix-It (BIFI) unsupervisedly learns a code fixer by alternating between repairing real bad code and generating realistic bad code with a learned breaker, guided by a critic. It achieves state-of-the-art repair accuracy on GitHub-Python and DeepFix without labeled data.

ABSTRACT

We consider repair tasks: given a critic (e.g., compiler) that assesses the quality of an input, the goal is to train a fixer that converts a bad example (e.g., code with syntax errors) into a good one (e.g., code with no syntax errors). Existing works create training data consisting of (bad, good) pairs by corrupting good examples using heuristics (e.g., dropping tokens). However, fixers trained on this synthetically-generated data do not extrapolate well to the real distribution of bad inputs. To bridge this gap, we propose a new training approach, Break-It-Fix-It (BIFI), which has two key ideas: (i) we use the critic to check a fixer's output on real bad inputs and add good (fixed) outputs to the training data, and (ii) we train a breaker to generate realistic bad code from good code. Based on these ideas, we iteratively update the breaker and the fixer while using them in conjunction to generate more paired data. We evaluate BIFI on two code repair datasets: GitHub-Python, a new dataset we introduce where the goal is to repair Python code with AST parse errors; and DeepFix, where the goal is to repair C code with compiler errors. BIFI outperforms existing methods, obtaining 90.5% repair accuracy on GitHub-Python (+28.5%) and 71.7% on DeepFix (+5.6%). Notably, BIFI does not require any labeled data; we hope it will be a strong starting point for unsupervised learning of various repair tasks.

Motivation & Objective

• Motivate learning a code fixer from unlabeled data using a critic to judge repair quality.
• Bridge the distribution gap between synthetic training data and real-world code errors.
• Develop a cyclical framework where a breaker and fixer mutually improve to produce realistic paired data.

Proposed method

• Initialize with synthetic bad/good pairs and train an initial fixer and breaker.
• Iteratively apply the fixer to real bad code and keep outputs verified as fixed by the critic (real paired data).
• Train the breaker on the real-pair data to generate realistic errors (on good code).
• Apply the breaker to good code and keep outputs that are actually broken (verified by the critic).
• Retrain the fixer on the combination of real-pairs and breaker-generated pairs to align with real error distributions.

Experimental results

Research questions

• RQ1 Can an unsupervised framework learn a code fixer from unlabeled code and a quality critic?
• RQ2 Does adversarial-like generation of realistic errors improve repair performance over purely synthetic data?
• RQ3 How does BIFI compare to backtranslation in unsupervised code repair?

Key findings

• BIFI outperforms the initial synthetic-data-trained fixers on GitHub-Python, achieving 90.5% repair accuracy (Round-2) versus 62.0% initially.
• BIFI yields a new state-of-the-art on DeepFix with 71.7% repair accuracy (Round-2) starting from DrRepair (66.1%).
• Using real bad code and breaker-generated bad code improves repair accuracy more than synthetic data alone (FixerOnly vs. BIFI).
• Backtranslation without the critic and real bad data is inferior to BIFI, with BIFI surpassing backtranslation by about 10% on GitHub-Python.