[Paper Review] Spillover of Antisocial Behavior from Fringe Platforms: The Unintended Consequences of Community Banning

Giuseppe Russo, Luca Verginer+2 more|arXiv (Cornell University)|Sep 20, 2022

Hate Speech and Cyberbullying Detection39 references4 citations

TL;DR

This study investigates the unintended consequence of banning antisocial online communities on mainstream platforms like Reddit, finding that users who migrate to fringe platforms with lax moderation exhibit increased toxicity and norm-violating behavior on Reddit due to co-participation. Using a difference-in-differences design on ~70,000 users from banned communities (r/The_Donald, r/GenderCritical, r/Incels), the authors show that prolonged engagement on fringe platforms significantly increases toxicity on Reddit, with effects intensifying over time and with higher activity levels.

ABSTRACT

Online platforms face pressure to keep their communities civil and respectful. Thus, the bannings of problematic online communities from mainstream platforms like Reddit and Facebook are often met with enthusiastic public reactions. However, this policy can lead users to migrate to alternative fringe platforms with lower moderation standards and where antisocial behaviors like trolling and harassment are widely accepted. As users of these communities often remain co-active across mainstream and fringe platforms, antisocial behaviors may spill over onto the mainstream platform. We study this possible spillover by analyzing around 70,000 users from three banned communities that migrated to fringe platforms: r/The_Donald, r/GenderCritical, and r/Incels. Using a difference-in-differences design, we contrast co-active users with matched counterparts to estimate the causal effect of fringe platform participation on users' antisocial behavior on Reddit. Our results show that participating in the fringe communities increases users' toxicity on Reddit (as measured by Perspective API) and involvement with subreddits similar to the banned community -- which often also breach platform norms. The effect intensifies with time and exposure to the fringe platform. In short, we find evidence for a spillover of antisocial behavior from fringe platforms onto Reddit via co-participation.

Motivation & Objective

• To examine whether banning antisocial communities on mainstream platforms leads to unintended behavioral spillovers onto the same platforms via co-active users.
• To assess the causal impact of participation in fringe platforms on users' antisocial behavior on Reddit after community bans.
• To evaluate how time and level of activity on fringe platforms influence the degree of behavioral spillover.
• To inform platform governance by identifying co-active users as a key vector for antisocial behavior spillover.
• To provide evidence-based recommendations for more effective moderation policies that avoid exacerbating toxicity.

Proposed method

• Employed a difference-in-differences (DiD) causal inference design comparing co-active users (active on both Reddit and fringe platforms) with matched Reddit-only users.
• Used longitudinal data from three banned Reddit communities: r/The_Donald, r/GenderCritical, and r/Incels, tracking ~70,000 users post-ban.
• Measured antisocial behavior on Reddit using Perspective API toxicity scores, a machine learning tool trained to detect toxic, disrespectful, or disruptive content.
• Constructed matched control groups of Reddit-only users based on pre-ban behavior and demographic proxies to ensure comparability.
• Tracked changes in toxicity and subreddits visited over time to assess the temporal dynamics of behavioral spillover.
• Accounted for potential mislabeling of co-active users due to username changes by analyzing the robustness of results under plausible assumptions.

Experimental results

Research questions

• RQ1 Does participation in fringe platforms following a community ban lead to increased antisocial behavior on the original mainstream platform?
• RQ2 How does the duration and level of activity on fringe platforms affect the magnitude of behavioral spillover on Reddit?
• RQ3 Are co-active users—who remain active on both platforms—more likely to exhibit norm-violating behavior on Reddit compared to users who only post on Reddit?
• RQ4 To what extent does the migration of banned communities to fringe platforms undermine the intended effects of community-level bans?
• RQ5 Can the behavioral spillover effect be mitigated by targeting co-active users in platform moderation strategies?

Key findings

• Co-active users who migrated to fringe platforms after banning exhibited significantly higher toxicity scores on Reddit compared to matched Reddit-only users.
• The increase in toxicity on Reddit was not immediate but intensified over time, indicating a cumulative effect of prolonged exposure to fringe platforms.
• Higher levels of activity on fringe platforms were strongly associated with greater increases in toxicity on Reddit, suggesting a dose-response relationship.
• Users who co-participated in banned communities and fringe platforms were more likely to engage with other subreddits that also violate platform norms.
• The results suggest that community-level bans may inadvertently strengthen antisocial ecosystems on fringe platforms and spread their toxicity back to mainstream platforms.
• The study’s findings represent a lower bound of the true effect, as mislabeling of co-active users (due to username changes) would only reduce observed effects if such users behaved similarly to correctly labeled ones.