[Paper Review] Evaluating software defect prediction performance: an updated benchmarking study

Libo Li, Stefan Lessmann+1 more|arXiv (Cornell University)|Jan 7, 2019

Software Engineering Research4 citations

TL;DR

This paper revisits software defect prediction by proposing an updated benchmarking framework that integrates improved evaluation metrics, testing procedures (frequentist vs. Bayesian), and diverse datasets and models. It finds that predictive accuracy is generally strong but highly sensitive to evaluation methodology, with classifier performance varying significantly across projects, underscoring the need for multidimensional assessment to reduce bias.

ABSTRACT

Accurately predicting faulty software units helps practitioners target faulty units and prioritize their efforts to maintain software quality. Prior studies use machine-learning models to detect faulty software code. We revisit past studies and point out potential improvements. Our new study proposes a revised benchmarking configuration. The configuration considers many new dimensions, such as class distribution sampling, evaluation metrics, and testing procedures. The new study also includes new datasets and models. Our findings suggest that predictive accuracy is generally good. However, predictive power is heavily influenced by the evaluation metrics and testing procedure (frequentist or Bayesian approach). The classifier results depend on the software project. While it is difficult to choose the best classifier, researchers should consider different dimensions to overcome potential bias.

Motivation & Objective

• To address limitations in prior software defect prediction studies by identifying methodological shortcomings in evaluation practices.
• To propose a revised benchmarking configuration that incorporates class distribution sampling, diverse evaluation metrics, and robust testing procedures (frequentist and Bayesian).
• To evaluate the performance of modern machine learning models on newly included datasets to ensure up-to-date and reliable benchmarking.
• To reduce bias in model selection by emphasizing multidimensional assessment across different software projects and configurations.
• To provide researchers with a standardized, comprehensive framework for fair and reproducible evaluation of defect prediction models.

Proposed method

• The study introduces a revised benchmarking configuration that includes systematic class distribution sampling to reflect real-world imbalanced data.
• It employs a range of evaluation metrics—such as AUC, F1-score, and G-mean—to assess model performance beyond accuracy alone.
• The authors apply both frequentist and Bayesian testing procedures to evaluate statistical significance and robustness of model comparisons.
• New datasets from public repositories are integrated to ensure relevance and diversity in benchmarking.
• Multiple machine learning models are evaluated under the same standardized configuration to ensure fair comparison.
• The framework emphasizes reproducibility by documenting all hyperparameters, data splits, and evaluation protocols.

Experimental results

Research questions

• RQ1 How do different evaluation metrics influence the perceived performance of defect prediction models?
• RQ2 What is the impact of using frequentist versus Bayesian testing procedures on model comparison outcomes?
• RQ3 How does model performance vary across different software projects under the same benchmarking configuration?
• RQ4 To what extent do class distribution sampling strategies affect model reliability and generalization?
• RQ5 Which combination of evaluation metrics and testing procedures provides the most robust and unbiased assessment of defect prediction models?

Key findings

• Predictive accuracy in software defect prediction is generally high, but performance varies significantly depending on the choice of evaluation metric.
• The Bayesian testing procedure reveals more nuanced differences in model performance compared to the frequentist approach, particularly in small-sample scenarios.
• Classifier performance is highly project-dependent, making it difficult to identify a single 'best' model across all contexts.
• The use of balanced evaluation metrics such as G-mean and F1-score provides more reliable insights than accuracy alone, especially in imbalanced datasets.
• The benchmarking framework demonstrates that methodological choices—especially in evaluation and testing—can introduce substantial bias in model assessment.
• The study confirms that no single model consistently outperforms others across all projects and evaluation criteria, highlighting the importance of context-aware model selection.