[Paper Review] Practical Program Repair in the Era of Large Pre-trained Language Models

Chunqiu Steven Xia, Yuxiang Wei+1 more|arXiv (Cornell University)|Oct 25, 2022

Software Engineering ResearchComputer Science29 citations

TL;DR

The paper conducts the first extensive evaluation of state-of-the-art large PLMs for automated program repair across multiple datasets and languages, demonstrating that PLMs can outperform traditional APR tools, with larger models generally performing better and infilling/suffix context improving patch quality.

ABSTRACT

Automated Program Repair (APR) aims to help developers automatically patch software bugs. However, current state-of-the-art traditional and learning-based APR techniques face the problem of limited patch variety, failing to fix complicated bugs. This is mainly due to the reliance on bug-fixing datasets to craft fix templates or directly predict potential patches. Large Pre-Trained Language Models (PLMs), trained using billions of text/code tokens, can potentially help avoid this issue. Very recently, researchers have directly leveraged PLMs for APR without relying on any bug-fixing datasets. Meanwhile, such existing work either failed to include state-of-the-art PLMs or was not evaluated on realistic datasets. In this work, we perform the first extensive study on directly applying PLMs for APR. We select 9 recent state-of-the-art PLMs, including both generative and infilling models, ranging from 125M to 20B in size. We designed 3 different repair settings to evaluate the different ways we can use PLMs to generate patches. We apply the PLMs under these repair settings on 5 datasets across 3 different languages and compare different PLMs in the number of bugs fixed, generation speed and compilation rate. Our study demonstrates that directly applying state-of-the-art PLMs can already substantially outperform all existing APR techniques on all our datasets. Among the studied PLMs, the scaling effect exists for APR where larger models tend to achieve better performance. Also, we show for the first time that suffix code after the buggy line (adopted in infilling-style APR) is important in not only generating more fixes but more patches with higher compilation rate. Besides patch generation, the PLMs consider correct patches to be more natural than other ones, and can even be leveraged for effective patch ranking or patch correctness checking.

Motivation & Objective

• Assess how various large PLMs perform on automated program repair across multiple datasets and languages.
• Compare PLM-based APR against state-of-the-art traditional and learning-based APR tools.
• Investigate how repair settings (complete function generation, infilling, single-line generation) affect patch quality and speed.
• Explore patch ranking and correctness checking using PLM-derived metrics like entropy.
• Identify practical guidelines to boost PLM-based APR performance (sample size, fix templates).

Proposed method

• Evaluate 9 large PLMs (125M–20B parameters), including generative and infilling models, on 5 real-world repair datasets in Java, Python, and C.
• Three repair settings: complete function generation, correct code infilling, and single-line generation.
• Use prompts and few-shot examples to enable PLMs to generate patches without bug-fix training data.
• Generate multiple patches per bug with nucleus sampling (top-p, temperature) and rank by patch entropy.
• Validate patches by running test suites to distinguish plausible from correct patches.
• Compare PLM-based APR with 20 baseline APR tools (learning-based and traditional).

Experimental results

Research questions

• RQ1 How do different types and sizes of PLMs perform under each APR setting across datasets and languages?
• RQ2 Do PLMs outperform state-of-the-art APR tools on real-world bugs?
• RQ3 Can PLMs be effectively used for patch ranking and correctness checking via entropy?
• RQ4 What strategies (more samples, incorporating fix templates) further boost PLM-based APR performance?

Key findings

• Larger PLMs generally yield more correct and plausible patches across datasets (scaling effect).
• Codex often outperforms other models in several settings due to code-focused pretraining and tuning.
• Infilling with suffix context (prefix+suffix) improves both the number of fixes and the patch compilation rate.
• Infilling models outperform generative counterparts on single-line and infilling tasks when suffix context is available.
• Using correct code infilling or single-line generation yields higher correct-to-plausible patch ratios than complete function generation.
• Patch generation speed decreases with larger models, but Codex demonstrates strong repair capability despite slower inference on some datasets.