[Paper Review] Run, Forest, Run? On Randomization and Reproducibility in Predictive Software Engineering

Cynthia C. S. Liem, Annibale Panichella|arXiv (Cornell University)|Dec 15, 2020

Software Engineering Research96 references4 citations

TL;DR

This paper investigates the impact of randomization on reproducibility in predictive software engineering, revealing widespread inconsistencies due to uncontrolled data sampling, random seeds, and library-specific implementations. It proposes five practical guidelines to enhance reproducibility through repeated runs, explicit parameter reporting, and statistical validation across multiple libraries and datasets.

ABSTRACT

Machine learning (ML) has been widely used in the literature to automate software engineering tasks. However, ML outcomes may be sensitive to randomization in data sampling mechanisms and learning procedures. To understand whether and how researchers in SE address these threats, we surveyed 45 recent papers related to three predictive tasks: defect prediction (DP), predictive mutation testing (PMT), and code smell detection (CSD). We found that less than 50% of the surveyed papers address the threats related to randomized data sampling (via multiple repetitions); only 8% of the papers address the random nature of ML; and parameter values are rarely reported (only 18% of the papers). To assess the severity of these threats, we conducted an empirical study using 26 real-world datasets commonly considered for the three predictive tasks of interest, considering eight common supervised ML classifiers. We show that different data resamplings for 10-fold cross-validation lead to extreme variability in observed performance results. Furthermore, randomized ML methods also show non-negligible variability for different choices of random seeds. More worryingly, performance and variability are inconsistent for different implementations of the conceptually same ML method in different libraries, as also shown through multi-dataset pairwise comparison. To cope with these critical threats, we provide practical guidelines on how to validate, assess, and report the results of predictive methods.

Motivation & Objective

• To assess how frequently researchers in predictive software engineering address randomization threats in data sampling, model training, and implementation choices.
• To investigate the extent of performance variability due to randomized data resampling and random seeds in common ML classifiers.
• To evaluate consistency of results across different ML libraries for the same algorithm and model configuration.
• To identify gaps in reporting practices for hyperparameters, random seeds, and library versions in recent SE literature.
• To develop and propose practical, actionable guidelines for improving reproducibility in predictive software engineering research.

Proposed method

• Conducted a systematic literature review of 45 recent papers across defect prediction, predictive mutation testing, and code smell detection to assess reporting of randomization and parameter settings.
• Performed an empirical study using 26 real-world SE datasets and eight supervised ML classifiers across three popular ML libraries (scikit-learn, Weka, R).
• Executed 100 repeated runs per dataset and model configuration to measure variability due to random seed choices and data resampling in 10-fold cross-validation.
• Compared results from identical model configurations implemented in different libraries to assess implementation consistency and default hyperparameter differences.
• Applied the Friedman test with post-hoc Nemenyi test to evaluate statistical significance of performance differences across models.
• Proposed five evidence-based guidelines for improving reproducibility, including repeated runs, explicit seed and parameter reporting, and awareness of solver and library behavior.

Experimental results

Research questions

• RQ1 To what extent do recent predictive SE papers report randomized data sampling procedures and multiple repetitions?
• RQ2 How much performance variability arises from different random seeds in ML training procedures across common classifiers?
• RQ3 How consistent are results across different ML library implementations of the same algorithm, despite identical conceptual configurations?
• RQ4 What is the impact of unreported or inconsistent hyperparameters and library versions on result reproducibility?
• RQ5 What statistical and reporting practices can improve the reliability and reproducibility of predictive SE results?

Key findings

• Less than 50% of surveyed papers addressed data resampling variability through multiple repetitions, indicating a major gap in methodological rigor.
• Only 8% of papers explicitly reported or controlled for random seeds in ML training, despite significant performance variability across different seeds.
• Performance results varied substantially across different implementations of the same ML model in different libraries, even with identical hyperparameter descriptions.
• Default parameter values—such as the number of trees in random forests—changed significantly across library versions, leading to inconsistent outcomes.
• Statistical analysis revealed that while Random Forests consistently ranked highly, they were not statistically superior to other models when corrected for multiple comparisons.
• The study demonstrated that default configurations across libraries differ widely, undermining reproducibility and necessitating explicit reporting of library names, versions, and parameter settings.