[Paper Review] Benchmarking cross-project defect prediction approaches with costs metrics

Steffen Herbold|arXiv (Cornell University)|Jan 12, 2018

Software Engineering Research64 references3 citations

TL;DR

This paper benchmarks 26 cross-project defect prediction (CPDP) approaches using cost-sensitive metrics, revealing that a trivial baseline—treating all code as defective—outperforms most CPDP methods in cost efficiency. The study shows no correlation between traditional performance metrics and cost-based rankings, indicating current CPDP models fail to deliver real cost savings despite strong performance on standard metrics.

ABSTRACT

Defect prediction can be a powerful tool to guide the use of quality assurance resources. In recent years, many researchers focused on the problem of Cross-Project Defect Prediction (CPDP), i.e., the creation of prediction models based on training data from other projects. However, only few of the published papers evaluate the cost efficiency of predictions, i.e., if they save costs if they are used to guide quality assurance efforts. Within this paper, we provide a benchmark of 26 CPDP approaches based on cost metrics. Our benchmark shows that trivially assuming everything as defective is on average better than CPDP under cost considerations. Moreover, we show that our ranking of approaches using cost metrics is uncorrelated to a ranking based on metrics that do not directly consider costs. These findings show that we must put more effort into evaluating the actual benefits of CPDP, as the current state of the art of CPDP can actually be beaten by a trivial approach in cost-oriented evaluations.

Motivation & Objective

• To evaluate the cost efficiency of 26 CPDP approaches using real-world cost metrics, addressing a gap in existing benchmarks that rely solely on machine learning performance metrics.
• To determine whether CPDP models actually reduce costs in practice, especially given that post-release defect fixes are 15–50× more expensive than pre-release reviews.
• To compare cost-based rankings of CPDP approaches with rankings based on traditional metrics like AUC and F-measure, assessing consistency across evaluation paradigms.
• To identify whether any CPDP approach significantly outperforms a trivial baseline (predicting all code as defective) under cost-sensitive evaluation.
• To highlight the threat of small sample sizes in defect prediction research and advocate for larger-scale benchmarks to improve generalizability.

Proposed method

• Replicated 26 CPDP approaches from 2008–2016 using the same data sets as Herbold et al. (2017a), ensuring methodological consistency.
• Applied three cost metrics—total cost, cost of false negatives, and cost of false positives—based on real cost ratios (e.g., 15–50× higher cost for post-release defects).
• Used statistical significance testing (e.g., Wilcoxon signed-rank test) to compare each CPDP approach against the trivial baseline and against each other.
• Evaluated models on 67 software projects from the JURECZKO and PROMISE data sets, with defect labels derived from commit histories and issue tracking systems.
• Performed sanity checks against prior benchmarks to ensure implementation correctness and construct validity.
• Used multiple data sets to reduce noise and mislabeling effects, enhancing reliability of cost-based comparisons.

Experimental results

Research questions

• RQ1 Does cross-project defect prediction (CPDP) outperform a trivial baseline that treats all code as defective when evaluated using cost metrics?
• RQ2 How does the ranking of CPDP approaches based on cost metrics compare to rankings based on traditional machine learning metrics like AUC and F-measure?
• RQ3 Which CPDP approaches achieve statistically significant cost advantages over the trivial baseline across different cost metrics?
• RQ4 To what extent do existing CPDP methods fail to deliver cost savings despite strong performance on standard evaluation metrics?
• RQ5 How does sample size affect the generalizability of CPDP benchmark results, and what are the implications for future research?

Key findings

• The trivial baseline—predicting all code as defective—performs better than the majority of CPDP approaches under cost metrics, indicating that current CPDP models do not reduce costs.
• Only one CPDP approach (Liu et al., 2010) significantly outperforms the trivial baseline on one of the two data sets, and only for one of the three cost metrics.
• There is no statistically significant correlation between rankings based on traditional metrics (e.g., AUC, F-measure) and rankings based on cost metrics, indicating that standard metrics are poor predictors of cost efficiency.
• The two best-performing CPDP methods—Liu et al. (2010) and Canfora et al. (2013, 2015)—are the only ones that explicitly optimize for cost during model training.
• The benchmark results suggest that optimizing for cost directly is essential, as models optimized only for prediction accuracy fail to deliver cost savings in practice.
• The small sample size of 67 projects is a major threat to external validity, limiting generalizability and raising concerns about a potential replication crisis in defect prediction research.