[Paper Review] Operationalizing Conflict and Cooperation between Automated Software Agents in Wikipedia: A Replication and Expansion of 'Even Good Bots Fight'

R. Stuart Geiger, Aaron Halfaker|arXiv (Cornell University)|Oct 16, 2018

Wikis in Education and Collaboration59 references7 citations

TL;DR

This paper replicates and extends the PLoS ONE study 'Even Good Bots Fight' by applying a mixed-methods approach—combining quantitative metrics with qualitative trace ethnography—to re-examine bot-bot interactions on Wikipedia. It finds that most interactions labeled as 'conflict' are actually routine, collaborative, or productive, challenging prior claims of widespread bot conflict and proposing new operational definitions and computational classification methods using edit summary patterns.

ABSTRACT

This paper replicates, extends, and refutes conclusions made in a study published in PLoS ONE ("Even Good Bots Fight"), which claimed to identify substantial levels of conflict between automated software agents (or bots) in Wikipedia using purely quantitative methods. By applying an integrative mixed-methods approach drawing on trace ethnography, we place these alleged cases of bot-bot conflict into context and arrive at a better understanding of these interactions. We found that overwhelmingly, the interactions previously characterized as problematic instances of conflict are typically better characterized as routine, productive, even collaborative work. These results challenge past work and show the importance of qualitative/quantitative collaboration. In our paper, we present quantitative metrics and qualitative heuristics for operationalizing bot-bot conflict. We give thick descriptions of kinds of events that present as bot-bot reverts, helping distinguish conflict from non-conflict. We computationally classify these kinds of events through patterns in edit summaries. By interpreting found/trace data in the socio-technical contexts in which people give that data meaning, we gain more from quantitative measurements, drawing deeper understandings about the governance of algorithmic systems in Wikipedia. We have also released our data collection, processing, and analysis pipeline, to facilitate computational reproducibility of our findings and to help other researchers interested in conducting similar mixed-method scholarship in other platforms and contexts.

Motivation & Objective

• To critically assess and replicate the PLoS ONE study claiming significant bot-bot conflict on Wikipedia.
• To challenge the validity of purely quantitative methods in identifying bot conflict by integrating qualitative, socio-technical context.
• To develop operational definitions of bot-bot conflict and cooperation grounded in real-world edit behaviors.
• To create a computationally reproducible pipeline for classifying bot interactions using edit summaries and trace data.
• To demonstrate the necessity of combining quantitative analysis with qualitative interpretation for understanding algorithmic governance in online platforms.

Proposed method

• Employing a mixed-methods design that integrates quantitative analysis with qualitative trace ethnography of Wikipedia edit histories.
• Using computational classification of bot edit summaries to distinguish types of bot interactions based on linguistic and behavioral patterns.
• Analyzing trace data from Wikipedia's revision history to identify and contextualize bot-bot reverts and edits.
• Applying qualitative heuristics to reinterpret quantitatively detected 'conflicts' within their socio-technical context.
• Developing and releasing a full data collection, processing, and analysis pipeline to ensure computational reproducibility.
• Drawing on socio-technical frameworks to interpret how meaning and intent are embedded in bot edit summaries and actions.

Experimental results

Research questions

• RQ1 To what extent do quantitative measures of bot-bot interaction on Wikipedia accurately reflect conflict, as opposed to routine or collaborative work?
• RQ2 How can qualitative context from edit summaries and revision histories improve the interpretation of bot interaction metrics?
• RQ3 What kinds of bot interactions are systematically misclassified as conflict by purely quantitative methods?
• RQ4 How can edit summaries be used to computationally classify bot interactions into conflict, cooperation, or neutral categories?
• RQ5 What methodological improvements are needed to study algorithmic governance in collaborative online systems?

Key findings

• The vast majority of bot-bot interactions previously labeled as 'conflict' were not adversarial but rather routine, productive, or collaborative in nature.
• Quantitative metrics alone overestimate bot conflict because they fail to account for context, intent, and edit summary semantics.
• Edit summaries contain reliable patterns that can computationally classify bot interactions into distinct behavioral categories with high interpretive fidelity.
• The study's mixed-methods approach revealed that many bot reverts were part of coordinated maintenance or cleanup efforts rather than disputes.
• The authors successfully operationalized conflict and cooperation through a combination of quantitative metrics and qualitative heuristics, offering a replicable framework for future research.
• The authors released a fully reproducible data and analysis pipeline to support further mixed-methods research on algorithmic systems in online communities.