[Paper Review] Revisiting Process versus Product Metrics: a Large Scale Analysis

Suvodeep Majumder, Pranav Mody+1 more|arXiv (Cornell University)|Aug 21, 2020

Software Engineering Research4 citations

TL;DR

This study re-evaluates the effectiveness of process versus product metrics for defect prediction using a large-scale analysis of 722,471 commits across 700 GitHub projects. It finds that process metrics significantly outperform product metrics (AUC: 95% vs. 54%, recall: 98% vs. 44%), but warns that metric importance rankings from small-scale studies do not generalize to large-scale settings, advocating for ensemble methods and in-the-large analysis to ensure robust, generalizable insights.

ABSTRACT

Numerous methods can build predictive models from software data. However, what methods and conclusions should we endorse as we move from analytics in-the-small (dealing with a handful of projects) to analytics in-the-large (dealing with hundreds of projects)? To answer this question, we recheck prior small-scale results (about process versus product metrics for defect prediction and the granularity of metrics) using 722,471 commits from 700 Github projects. We find that some analytics in-the-small conclusions still hold when scaling up to analytics in-the-large. For example, like prior work, we see that process metrics are better predictors for defects than product metrics (best process/product-based learners respectively achieve recalls of 98\%/44\% and AUCs of 95\%/54\%, median values). That said, we warn that it is unwise to trust metric importance results from analytics in-the-small studies since those change dramatically when moving to analytics in-the-large. Also, when reasoning in-the-large about hundreds of projects, it is better to use predictions from multiple models (since single model predictions can become confused and exhibit a high variance).

Motivation & Objective

• To assess whether conclusions from small-scale software analytics (analytics in-the-small) hold when scaled to hundreds of projects (analytics in-the-large).
• To evaluate the relative performance of process metrics versus product metrics in defect prediction across a large corpus of open-source projects.
• To investigate whether metric importance rankings derived from small-scale studies remain stable when applied to large-scale data.
• To recommend best practices for industrial and academic use of predictive models in software engineering, emphasizing scalability and generalizability.

Proposed method

• Collected and analyzed 722,471 commits from 700 GitHub projects using both process and product metrics.
• Applied multiple machine learning models—Random Forest, Logistic Regression, and SVM—for defect prediction.
• Used hyperparameter optimization (HPO) to tune models and ensure fair comparison across learners.
• Compared model performance using standard metrics: AUC, recall, and F1-score to evaluate predictive power.
• Conducted ablation studies to assess the impact of individual metrics and their importance rankings.
• Employed ensemble modeling (e.g., Random Forest) to reduce variance and improve stability in large-scale predictions.

Experimental results

Research questions

• RQ1 Do process metrics consistently outperform product metrics for defect prediction in large-scale software analytics?
• RQ2 How stable are metric importance rankings derived from small-scale studies when applied to large-scale data?
• RQ3 What are the practical implications of using single-model versus ensemble predictions in large-scale software defect prediction?
• RQ4 How do systems and data collection challenges differ between process and product metrics at scale?
• RQ5 Can findings from small-scale studies be reliably generalized to large-scale software projects?

Key findings

• Process metrics achieved a median AUC of 95% and recall of 98% in defect prediction, significantly outperforming product metrics, which achieved a median AUC of 54% and recall of 44%.
• Metric importance rankings from small-scale studies (e.g., ICSE 2013, ICSM 2010) do not generalize to large-scale data, with top-ranked metrics in small studies often becoming irrelevant in large-scale analysis.
• Single-model predictions in large-scale settings exhibit high variance and are prone to confusion, making ensemble methods like Random Forest more reliable.
• Product metrics require significantly more computational resources—approximately 500 CPU days—compared to process metrics, which are an order of magnitude faster to compute.
• The study confirms that process metrics are more scalable and practical for industrial use due to lower collection overhead and language-agnostic applicability.
• Findings from small-scale qualitative studies do not generalize; therefore, quantitative replication across large project sets is essential to derive stable, generalizable insights.