[Paper Review] Don't Disturb Me: Challenges of Interacting with SoftwareBots on Open Source Software Projects

Mairieli Wessel, Igor Wiese+2 more|arXiv (Cornell University)|Mar 25, 2021

Wikis in Education and Collaboration77 references40 citations

TL;DR

This paper investigates the challenges of interacting with software bots in open source software (OSS) projects, particularly on GitHub pull requests. Through semi-structured interviews with 21 practitioners, the authors identify 'noise'—caused by excessive, verbose, or disruptive bot behaviors—as the central problem disrupting developer communication and workflow. The key contribution is a theory of how bot-induced noise leads to information overload, offering insights for improving human-bot interaction design in collaborative development environments.

ABSTRACT

Software bots are used to streamline tasks in Open Source Software (OSS) projects' pull requests, saving development cost, time, and effort. However, their presence can be disruptive to the community. We identified several challenges caused by bots in pull request interactions by interviewing 21 practitioners, including project maintainers, contributors, and bot developers. In particular, our findings indicate noise as a recurrent and central problem. Noise affects both human communication and development workflow by overwhelming and distracting developers. Our main contribution is a theory of how human developers perceive annoying bot behaviors as noise on social coding platforms. This contribution may help practitioners understand the effects of adopting a bot, and researchers and tool designers may leverage our results to better support human-bot interaction on social coding platforms.

Motivation & Objective

• To understand the challenges introduced by software bots in pull request interactions within open source software (OSS) projects.
• To investigate how bot behaviors affect human communication and development workflow in collaborative coding environments.
• To identify and analyze the perception of bot behaviors as 'noise' by developers, including maintainers, contributors, and bot developers.
• To develop a theory grounded in practitioner experiences explaining how bot interactions disrupt workflow and communication.

Proposed method

• Conducted 21 semi-structured interviews with experienced practitioners: OSS maintainers, contributors, and bot developers.
• Used qualitative analysis with constant comparison to identify recurring themes and patterns in bot-related challenges.
• Validated findings through member-checking with interview participants to ensure accuracy and reliability.
• Theoretical development focused on noise as a central construct, derived from interview data, to explain disruptive bot behaviors.
• Focused on GitHub-based interactions, analyzing how bot actions affect communication flow and cognitive load.
• Explored the impact of bot verbosity, frequency of actions, and unrequested interventions on developer experience.

Experimental results

Research questions

• RQ1 RQ1: What interaction challenges do software bots introduce when supporting pull requests in open source software projects?
• RQ2 RQ2: How do developers perceive and respond to bot behaviors that disrupt communication and workflow?
• RQ3 RQ3: What role does 'noise' play in the negative perception of software bots in collaborative development settings?

Key findings

• Developers frequently perceive bot behaviors—such as excessive comments, repetitive notifications, and verbose feedback—as noise that disrupts communication and workflow.
• Noise from bots leads to information overload, causing developers to feel overwhelmed and distracted, even when bots are intended to reduce workload.
• Maintainers often respond to noise by reconfiguring bots, redesigning interactions, or disabling notifications, including unwatching repositories.
• Newcomers are especially affected by bot feedback, which can feel discouraging due to the high volume of critical remarks without supportive guidance.
• Despite bots reducing the number of human comments, developers often do not perceive this reduction, leading to expectation mismatches.
• The study identifies a theoretical link between bot-induced noise and cognitive disruption, highlighting the need for more thoughtful bot design in social coding platforms.