[Paper Review] Content Removal as a Moderation Strategy: Compliance and Other Outcomes in the ChangeMyView Community

Kumar Bhargav Srinivasan, Cristian Danescu-Niculescu-Mizil+2 more|arXiv (Cornell University)|Oct 21, 2019

Social Media and PoliticsSocial Sciences42 references33 citations

TL;DR

This paper investigates whether deleting moderator-removed comments in CMV causally changes the author’s future behavior, using delayed feedback and interrupted time-series designs to separate removal effects from posting context.

ABSTRACT

Moderators of online communities often employ comment deletion as a tool. We ask here whether, beyond the positive effects of shielding a community from undesirable content, does comment removal actually cause the behavior of the comment's author to improve? We examine this question in a particularly well-moderated community, the ChangeMyView subreddit. The standard analytic approach of interrupted time-series analysis unfortunately cannot answer this question of causality because it fails to distinguish the effect of having made a non-compliant comment from the effect of being subjected to moderator removal of that comment. We therefore leverage a "delayed feedback" approach based on the observation that some users may remain active between the time when they posted the non-compliant comment and the time when that comment is deleted. Applying this approach to such users, we reveal the causal role of comment deletion in reducing immediate noncompliance rates, although we do not find evidence of it having a causal role in inducing other behavior improvements. Our work thus empirically demonstrates both the promise and some potential limits of content removal as a positive moderation strategy, and points to future directions for identifying causal effects from observational data.

Motivation & Objective

• Evaluate whether moderator-removed comments lead to improved future behavior by affected authors in CMV.
• Develop causal inference methods suitable for observational moderation data (delayed feedback design).
• Distinguish between effects of content removal and effects of posting the problematic comment itself.

Proposed method

• Apply interrupted time-series (ITS) analysis to compare pre- and post-removal behavior in non-affected trees.
• Introduce the delayed feedback (DF) design to separate the effect of removal from the effects of posting the problematic comment.
• Use a temporally paired matched-control approach to mitigate temporal confounds when removal is delayed.
• Operate on a large CMV dataset with post trees, removed comments, and moderator metadata collected from 2013–2018.
• Define and compute a set of comment- and user-level features to test five behavioral hypotheses (noncompliance, toxicity, achievement, engagement).

Experimental results

Research questions

• RQ1 Does comment removal causally reduce subsequent rule violations by affected individuals who continue to participate in CMV?
• RQ2 Does removal causally reduce toxicity in subsequent comments by the affected individuals?
• RQ3 Does removal causally improve achievement or community approval of subsequent contributions?
• RQ4 Does removal causally affect subsequent engagement levels of affected individuals?
• RQ5 How well can delayed feedback designs disentangle the causal effect of moderation from temporal/contextual factors in observational data?

Key findings

• Comment deletion causally reduces immediate noncompliance rates in the subset of users who remain active after removal (via the delayed feedback design).
• Interrupted time-series analysis shows some significant pre/post changes in behavior, but these cannot be attributed to the causal effect of removal for most outcomes.
• Removal does not provide evidence of a causal role in improving other behaviors beyond reducing near-term noncompliance.
• Affected individuals who continue to participate often abandon CMV at higher rates, complicating causal attribution across all outcomes.
• The delayed-feedback approach generally supports the promise of content removal as positive moderation, while highlighting its limits for inducing broader behavioral improvements.