[Paper Review] Pull Request Decision Explained: An Empirical Overview

Xunhui Zhang, Yue Yu+2 more|arXiv (Cornell University)|May 28, 2021

Software Engineering Research60 references4 citations

TL;DR

This study empirically explains pull request (PR) decision-making on GitHub using a large-scale dataset of 3.35 million PRs from 11,230 projects. It identifies that the most critical factor is whether the contributor and integrator are the same person, explaining over 30% of decision variance, with context-dependent influences from CI usage and discussion sentiment.

ABSTRACT

Context: Pull-based development model is widely used in open source, leading the trends in distributed software development. One aspect which has garnered significant attention is studies on pull request decision - identifying factors for explanation. Objective: This study builds on a decade long research on pull request decision to explain it. We empirically investigate how factors influence pull request decision and scenarios that change the influence of factors. Method: We identify factors influencing pull request decision on GitHub through a systematic literature review and infer it by mining archival data. We collect a total of 3,347,937 pull requests with 95 features from 11,230 diverse projects on GitHub. Using this data, we explore the relations of the factors to each other and build mixed-effect logistic regression models to empirically explain pull request decision. Results: Our study shows that a small number of factors explain pull request decision with the integrator same or different from the submitter as the most important factor. We also noted that some factors are important only in special cases e.g., the percentage of failed builds is important for pull request decision when continuous integration is used.

Motivation & Objective

• To synthesize existing knowledge on factors influencing pull request decisions in open-source software development.
• To identify and quantify the most influential factors in PR decision-making using empirical data from GitHub.
• To investigate how the importance of factors changes across different development contexts, such as CI usage or comment presence.
• To provide a comprehensive, diverse dataset of 11,230 projects and 95 features for future research in software engineering.
• To build mixed-effect logistic regression models that explain PR acceptance or rejection with high empirical validity.

Proposed method

• Conducted a systematic literature review (SLR) to identify 95 factors influencing PR decisions from existing research.
• Collected and processed archival data from 11,230 GitHub projects, including PR metadata, contributor behavior, CI results, and discussion features.
• Preprocessed data to extract 95 indicators representing the identified factors, ensuring consistency across diverse projects and programming languages.
• Applied mixed-effect logistic regression models to analyze the influence of each factor on PR decision (acceptance/rejection), accounting for project-level random effects.
• Performed subgroup analyses to assess how factor importance shifts in specific contexts (e.g., CI-enabled PRs, PRs with comments).
• Validated findings through exploratory data analysis and model diagnostics to ensure robustness and generalizability.

Experimental results

Research questions

• RQ1 RQ1: How do various factors influence pull request decision outcomes?
• RQ2 RQ2: Do the influences of these factors change depending on contextual conditions (e.g., CI usage, comment activity)?

Key findings

• The factor 'contributor and integrator are the same user' is the most influential, explaining over 30% of the variance in PR decision outcomes.
• When continuous integration (CI) is used, the percentage of failed CI builds becomes a significant predictor of PR rejection.
• Positive sentiment in PR discussions increases the likelihood of acceptance, especially when comments are present.
• The use of CI tools reduces the relative importance of comments in decision-making, suggesting CI can substitute for peer review in some cases.
• Only 5 to 10 factors account for the majority of the explanatory power in PR decision models, indicating high factor efficiency.
• The study identifies a previously overlooked but highly significant factor—contributor-integrator identity—absent in prior research.