[Paper Review] Developers' Perception of GitHub Actions: A Survey Analysis

Sk Golam Saroar, Maleknaz Nayebi|arXiv (Cornell University)|Mar 7, 2023

Scientific Computing and Data ManagementDecision Sciences3 citations

TL;DR

This study investigates developers' perceptions of GitHub Actions through a survey of 90 developers and users, revealing key motivations, challenges, and best practices in using, developing, and debugging Actions. It finds that developers prioritize verified creators and high star ratings, struggle significantly with YAML configuration, and often avoid Actions due to complexity and security concerns, despite their automation benefits.

ABSTRACT

GitHub introduced "Actions" in 2019 to increase workflow velocity and add customized automation to the repositories. Any individual can develop Actions for automating workflow on GitHub repositories and others can reuse them whenever open source. GitHub introduced its marketplace for commercializing and sharing these automation tools, which currently hosts 16,730 Actions. Yet, there are numerous Actions that are developed and distributed in local repositories and outside the Marketplace. So far, the research community conducted mining studies to understand Actions with a significant focus on CI/CD. We conducted a survey study with 90 Action developers and users to understand the motivations and best practices in using, developing, and debugging Actions, and the challenges associated with these tasks. We found that developers prefer Actions with verified creators and more stars when choosing between similar Actions, and often switch to an alternative Action when facing bugs or a lack of documentation. 60.87% of the developers consider the composition of YAML files, which are essential for Action integration, challenging and error-prone. They mainly check Q\&A forums to fix issues with these YAML files. Finally, developers tend to avoid using Actions (and hence automation) to reduce complexity, and security risk, or when the benefits of Actions are not worth the cost/effort of setting up Actions for automation.

Motivation & Objective

• To understand developers' motivations for using, developing, and debugging GitHub Actions.
• To identify the challenges developers face when integrating and maintaining Actions in their workflows.
• To explore the criteria developers use when selecting Actions, especially regarding trust and reliability.
• To examine the reasons behind the avoidance of Actions despite their automation potential.

Proposed method

• A survey was administered to 90 developers and users of GitHub Actions, including both creators and consumers of Actions.
• The survey collected data on preferences, challenges, debugging practices, and decision-making criteria when selecting Actions.
• Quantitative and qualitative responses were analyzed to identify patterns in developer behavior and perception.
• Focus was placed on understanding YAML configuration difficulties, trust indicators (e.g., stars, verification), and avoidance behaviors.

Experimental results

Research questions

• RQ1 What motivates developers to choose one GitHub Action over another, and which trust indicators do they prioritize?
• RQ2 What are the primary challenges developers face when using, developing, or debugging GitHub Actions?
• RQ3 How do developers resolve issues related to YAML configuration in Actions?
• RQ4 Why do developers sometimes avoid using GitHub Actions despite their automation benefits?

Key findings

• 60.87% of developers find the composition of YAML files for Actions to be challenging and error-prone, often requiring external help to resolve.
• Developers prefer Actions with verified creators and higher star counts when choosing between similar options, indicating trust in social and verification signals.
• When faced with bugs or insufficient documentation, 60.87% of developers switch to alternative Actions, highlighting the importance of reliability and clarity.
• A significant number of developers avoid using Actions altogether due to perceived complexity, security risks, or the high setup cost relative to benefits.