[Paper Review] On Distribution Shift in Learning-based Bug Detectors

Jingxuan He, Luca Beurer-Kellner+1 more|arXiv (Cornell University)|Apr 21, 2022

Software Engineering Research4 citations

TL;DR

This paper identifies distribution shift—specifically, the mismatch between synthetic bugs used in training and real bugs in practice—as the root cause of poor performance in learning-based bug detectors. The authors propose a two-phase training method: first pre-training on synthetic bugs to learn general features, then fine-tuning on a small, realistic dataset of real bugs with data imbalance, using focal loss, contrastive learning, and a multi-task hierarchy. The approach achieves high precision (up to 51%) on real-world test sets, significantly outperforming prior methods.

ABSTRACT

Deep learning has recently achieved initial success in program analysis tasks such as bug detection. Lacking real bugs, most existing works construct training and test data by injecting synthetic bugs into correct programs. Despite achieving high test accuracy (e.g., 90%), the resulting bug detectors are found to be surprisingly unusable in practice, i.e., <10% precision when used to scan real software repositories. In this work, we argue that this massive performance difference is caused by a distribution shift, i.e., a fundamental mismatch between the real bug distribution and the synthetic bug distribution used to train and evaluate the detectors. To address this key challenge, we propose to train a bug detector in two phases, first on a synthetic bug distribution to adapt the model to the bug detection domain, and then on a real bug distribution to drive the model towards the real distribution. During these two phases, we leverage a multi-task hierarchy, focal loss, and contrastive learning to further boost performance. We evaluate our approach extensively on three widely studied bug types, for which we construct new datasets carefully designed to capture the real bug distribution. The results demonstrate that our approach is practically effective and successfully mitigates the distribution shift: our learned detectors are highly performant on both our test set and the latest version of open source repositories. Our code, datasets, and models are publicly available at https://github.com/eth-sri/learning-real-bug-detector.

Motivation & Objective

• To identify the root cause of the performance gap between synthetic-bug-trained bug detectors and their real-world deployment.
• To address the fundamental distribution shift between synthetic bugs used in training and real bugs found in production code.
• To develop a practical, two-phase training framework that improves detector performance on real-world code repositories.
• To construct realistic datasets capturing the true distribution of real bugs, including extreme class imbalance.
• To demonstrate that combining pre-training on synthetic bugs with fine-tuning on real bugs yields significantly more effective and precise detectors.

Proposed method

• Propose a two-phase training pipeline: first pre-training on a balanced dataset of synthetic bugs, then fine-tuning on a realistic, imbalanced dataset of real bugs.
• Integrate a multi-task hierarchy to improve generalization and feature learning across related bug types.
• Apply focal loss to mitigate class imbalance during fine-tuning, focusing learning on hard-to-detect real bugs.
• Use contrastive learning to distinguish between buggy and non-buggy code pairs, enhancing representation learning.
• Leverage pre-trained code models (e.g., CuBERT) as the backbone network for code representation.
• Construct new datasets with real bugs extracted from GitHub commits and non-buggy code to reflect real-world data distribution.

Experimental results

Research questions

• RQ1 What causes the drastic drop in precision of learning-based bug detectors when deployed on real-world code repositories?
• RQ2 To what extent does distribution shift—between synthetic and real bug distributions—undermine the performance of existing bug detectors?
• RQ3 Can a two-phase training strategy, combining synthetic and real bug data, effectively mitigate distribution shift in bug detection?
• RQ4 How do auxiliary components like focal loss, contrastive learning, and task hierarchy improve detector performance on real-world data?
• RQ5 Can a detector trained on a small number of real bugs generalize effectively to large-scale, imbalanced production code?

Key findings

• The performance of existing bug detectors drops from over 90% precision on synthetic test sets to less than 10% on real-world test sets, demonstrating a severe distribution shift.
• On a real-world test set with 0.5% real bugs and 99.5% non-buggy code, the best baseline model achieves only 3% precision, highlighting the practical ineffectiveness of current approaches.
• The proposed two-phase training method achieves up to 51% precision on the same real-world test set, significantly outperforming baselines and demonstrating effective mitigation of distribution shift.
• The combination of pre-training on synthetic bugs and fine-tuning on real bugs with focal loss and contrastive learning leads to the highest performance, showing the necessity of both phases.
• The method generalizes across multiple bug types, including variable misuse and wrong binary operators, and performs well on the latest version of open-source repositories.
• The authors release public datasets, models, and code, enabling reproducibility and further research in realistic bug detection.