[Paper Review] Replication Can Improve Prior Results: A GitHub Study of Pull Request Acceptance

Di Chen, Kathyrn T. Stolee+1 more|arXiv (Cornell University)|Feb 9, 2019

Mobile Crowdsensing and Crowdsourcing52 references4 citations

TL;DR

This study uses crowdsourcing and data mining to partially replicate and extend a qualitative study on GitHub pull request acceptance. By applying insights from the original study to a larger dataset of 190 pull requests, the authors built a predictor with 90% F1 score—significantly outperforming the original study’s qualitative factors—demonstrating that replication with mixed methods enhances prior results.

ABSTRACT

Crowdsourcing and data mining can be used to effectively reduce the effort associated with the partial replication and enhancement of qualitative studies. For example, in a primary study, other researchers explored factors influencing the fate of GitHub pull requests using an extensive qualitative analysis of 20 pull requests. Guided by their findings, we mapped some of their qualitative insights onto quantitative questions. To determine how well their findings generalize, we collected much more data (170 additional pull requests from 142 GitHub projects). Using crowdsourcing, that data was augmented with subjective qualitative human opinions about how pull requests extended the original issue. The crowd's answers were then combined with quantitative features and, using data mining, used to build a predictor for whether code would be merged. That predictor was far more accurate that one built from the primary study's qualitative factors (F1=90 vs 68\%), illustrating the value of a mixed-methods approach and replication to improve prior results. To test the generality of this approach, the next step in future work is to conduct other studies that extend qualitative studies with crowdsourcing and data mining.

Motivation & Objective

• To test whether partial replication using crowdsourcing and data mining can improve the accuracy of predictions from a primary qualitative study on pull request acceptance.
• To evaluate the external validity of findings from a prior qualitative study by scaling them to a larger dataset of 190 pull requests.
• To compare the predictive performance of features derived from qualitative insights (via crowdsourcing) against quantitative features from the literature.
• To demonstrate the value of a mixed-methods approach in software engineering research, combining expert insights with scalable data collection and analysis.
• To establish a replicable methodology for extending qualitative studies through cost-effective, large-scale data augmentation and data mining.

Proposed method

• The study used insights from a prior qualitative study (TDH) to design micro-task questions for crowdsourcing, focusing on pull request discussion features and their impact on acceptance.
• Crowd workers on MTurk evaluated 170 additional pull requests beyond the original 20, providing subjective assessments of how requests extended original issues.
• The crowd-sourced responses were combined with quantitative metadata (e.g., number of comments, time to merge) to form a comprehensive dataset.
• Data mining techniques were applied to build predictive models for pull request acceptance using both crowd-derived features and literature-based quantitative features.
• Quality control was ensured via golden questions and result aggregation across multiple workers to improve reliability.
• Predictor performance was evaluated using F1 score, comparing models trained on crowd-annotated features versus those based on traditional quantitative metrics.

Experimental results

Research questions

• RQ1 Can a partial replication of a qualitative study on pull request acceptance be effectively scaled using crowdsourcing?
• RQ2 Do qualitative insights from a small sample generalize to a larger dataset when validated through crowdsourced evaluation?
• RQ3 Can data mining on crowd-sourced annotations improve prediction accuracy beyond models based on quantitative features alone?
• RQ4 How do the predictive performances of models based on qualitative insights (via crowdsourcing) compare to those based on established quantitative metrics?
• RQ5 To what extent can mixed-methods replication enhance the robustness and accuracy of findings from primary qualitative studies?

Key findings

• The predictor built using crowd-sourced qualitative features achieved an F1 score of 90%, significantly outperforming the 68% F1 score of the predictor based on the original study’s qualitative factors.
• The model trained on quantitative features from the literature outperformed the model based solely on the original qualitative insights, but the crowd-enhanced model still showed strong performance.
• Crowdsourcing successfully replicated key findings from the original qualitative study, indicating stability and external validity of the insights when scaled.
• The study confirmed that pull request acceptance is influenced by factors such as the submitter’s prior interaction level and discussion dynamics, which were consistently observed across the larger dataset.
• Data mining on crowd-annotated data revealed new, actionable predictors that were not apparent from qualitative analysis alone, demonstrating the value of mixed-methods scaling.
• The results support the conjecture that combining crowdsourcing with data mining enhances the accuracy and robustness of findings from primary qualitative studies in software engineering.