[Paper Review] How Do Software Developers Use GitHub Actions to Automate Their Workflows?

Timothy Kinsman, Mairieli Wessel+2 more|arXiv (Cornell University)|Mar 22, 2021

Software Engineering Research38 references65 citations

TL;DR

This study investigates how open-source developers use GitHub Actions to automate workflows, employing a Regression Discontinuity Design across 926 projects. It finds that adoption increases rejected pull requests and reduces commits in merged pull requests, indicating potential workflow friction despite positive developer perception.

ABSTRACT

Automated tools are frequently used in social coding repositories to perform repetitive activities that are part of the distributed software development process. Recently, GitHub introduced GitHub Actions, a feature providing automated workflows for repository maintainers. Although several Actions have been built and used by practitioners, relatively little has been done to evaluate them. Understanding and anticipating the effects of adopting such kind of technology is important for planning and management. Our research is the first to investigate how developers use Actions and how several activity indicators change after their adoption. Our results indicate that, although only a small subset of repositories adopted GitHub Actions to date, there is a positive perception of the technology. Our findings also indicate that the adoption of GitHub Actions increases the number of monthly rejected pull requests and decreases the monthly number of commits on merged pull requests. These results are especially relevant for practitioners to understand and prevent undesirable effects on their projects.

Motivation & Objective

• To understand how open-source projects use GitHub Actions for workflow automation.
• To examine developer discussions around GitHub Actions in issue trackers.
• To assess the impact of GitHub Actions adoption on project activity indicators such as pull request volume, commit frequency, and review cycles.
• To identify potential unintended consequences of automation on contributor engagement and code quality.

Proposed method

• Collected data from 3,190 active GitHub repositories using the RepoReapers dataset and GitHub API.
• Identified GitHub Actions adoption by detecting YAML files in the .github/workflows/ directory.
• Conducted manual analysis of 209 GitHub issues discussing Actions to categorize developer sentiment and concerns.
• Applied Regression Discontinuity Design (RDD) to model changes in project activity indicators before and after Actions adoption.
• Used statistical controls and placebo tests to ensure model robustness and reduce confounding factors.
• Tracked commit history of workflow files to analyze evolution and stability of Actions over time.

Experimental results

Research questions

• RQ1 RQ1: How do OSS projects use GitHub Actions, and how common is their adoption?
• RQ2 RQ2: How is the use of GitHub Actions discussed by developers in issue trackers?
• RQ3 RQ3: What is the impact of GitHub Actions adoption on project activity indicators such as pull request volume, commit frequency, and time-to-close?

Key findings

• Only 0.7% of 416,266 repositories studied adopted GitHub Actions, indicating limited but growing usage.
• Seventy percent of Actions were used in continuous integration, utilities, and deployment categories.
• Developer discussions on GitHub issues were predominantly positive, with common themes including migration from other tools and feature requests.
• Adoption of GitHub Actions led to a statistically significant increase in the number of monthly rejected pull requests.
• Adoption also led to a decrease in the number of commits per merged pull request, suggesting reduced code iteration post-merge.
• The study found no significant change in time-to-close pull requests, indicating that automation did not accelerate or delay resolution times.