[Paper Review] Early Life Cycle Software Defect Prediction. Why? How?

N. C. Shrikanth, Suvodeep Majumder+1 more|arXiv (Cornell University)|Nov 26, 2020

Software Engineering Research71 references4 citations

TL;DR

This paper proposes a data-lite approach to early life cycle software defect prediction, showing that models trained on just the first 150 commits and four months of data perform as well as those trained on full project histories. The key finding is that for long-running GitHub projects, 96% of the time, further data collection is unnecessary, challenging the prevailing 'more data is better' assumption in software analytics.

ABSTRACT

Many researchers assume that, for software analytics, "more data is better." We write to show that, at least for learning defect predictors, this may not be true. To demonstrate this, we analyzed hundreds of popular GitHub projects. These projects ran for 84 months and contained 3,728 commits (median values). Across these projects, most of the defects occur very early in their life cycle. Hence, defect predictors learned from the first 150 commits and four months perform just as well as anything else. This means that, at least for the projects studied here, after the first few months, we need not continually update our defect prediction models. We hope these results inspire other researchers to adopt a "simplicity-first" approach to their work. Some domains require a complex and data-hungry analysis. But before assuming complexity, it is prudent to check the raw data looking for "short cuts" that can simplify the analysis.

Motivation & Objective

• To challenge the assumption that more data always improves defect prediction models in software analytics.
• To investigate whether early-stage data can yield effective defect predictors without requiring long-term data collection.
• To evaluate whether data-hungry methods are necessary or if simpler, early-data approaches suffice for defect prediction.
• To provide a replicable methodology for assessing early-life cycle defect prediction using minimal data.
• To inspire a 'simplicity-first' approach in software engineering research, especially when data collection is costly or unstable.

Proposed method

• The study analyzes 155 long-running, high-star GitHub projects (median 84-month lifespan, 3,728 commits) to assess defect distribution across the project lifecycle.
• It compares defect prediction models trained on early data (first 150 commits and 4 months) versus models trained on full project histories.
• Six classification learners (Logistic Regression, Nearest Neighbor, Decision Tree, SVM, Random Forest, Naïve Bayes) are used to evaluate model performance across different training data windows.
• The evaluation uses seven metrics: Recall, PF, IFA, Brier, GM, D2H, and AUC, with precision excluded due to issues in unbalanced data.
• A sampling policy 'E' randomly selects 50 commits from the first 150 commits to ensure robustness against variability in early data.
• The approach compares models trained on early data (E) versus recent data (RR), assessing whether recency improves performance.

Experimental results

Research questions

• RQ1 Does training defect prediction models on early project data (first 4% of lifetime) perform as well as using full project histories?
• RQ2 Is there a point in the project lifecycle beyond which additional data provides diminishing returns for defect prediction?
• RQ3 Do recency-based models trained on recent data outperform models trained on early data?
• RQ4 Can a 'simplicity-first' approach reduce the need for continual model retraining without sacrificing accuracy?
• RQ5 To what extent can early data alone support reliable defect prediction in non-trivial, long-lived software projects?

Key findings

• Defects are highly concentrated in the early lifecycle: 96% of the project's lifetime does not require additional data for effective defect prediction.
• Models trained on the first 150 commits and four months of data perform as well as those trained on full project histories, indicating that early data is sufficient.
• There is no significant performance difference between models trained on early data and those trained on recent data, challenging the assumption that recency improves prediction.
• The performance of early-life cycle predictors is stable and comparable across six different learners, indicating robustness to model choice.
• The results suggest that data-hungry methods may be unnecessary for defect prediction in long-running, non-trivial projects, especially when early data is already informative.
• The study's findings call into question prior research that relies on recent validation data, as such data may lie in an uninformative region of the project lifecycle.