[Paper Review] WAC: A Corpus of Wikipedia Conversations for Online Abuse Detection

Noé Cécillon, Vincent Labatut|arXiv (Cornell University)|Jan 1, 2019

Hate Speech and Cyberbullying Detection16 references4 citations

TL;DR

This paper introduces WAC, a large-scale, publicly available corpus of 383,000 annotated Wikipedia conversation messages spanning 193,000 discussion threads, with high-quality abuse labels. It proposes a reconstruction pipeline combining the Wikipedia Comment Corpus and revision history to preserve conversational context, and introduces a benchmarking platform to enable fair comparison of abuse detection methods, demonstrating the utility of context-aware models through baseline evaluations using the Google Perspective API and a hybrid text- and graph-based classifier.

ABSTRACT

Figures used in "WAC: A Corpus of Wikipedia Conversations for Online Abuse Detection".https://www.aclweb.org/anthology/2020.lrec-1.173/<br>

Motivation & Objective

• To address the lack of large-scale, publicly available corpora of full Wikipedia conversation threads annotated for online abuse.
• To improve the replicability and fair comparison of abuse detection methods by introducing a shared benchmarking platform.
• To enable context-based abuse detection research by preserving conversational structure and dynamics in the dataset.
• To provide a high-quality, scalable resource for training and evaluating both content-based and structure-based abuse detection models.

Proposed method

• Reconstruct full conversation threads by linking messages from the Wikipedia Comment Corpus to their corresponding revision histories using unique identifiers (rev_id and page_id).
• Apply a multi-step pipeline to extract message-level annotations, including message type, author, talk page, and abuse labels from human-annotated data.
• Use a hybrid classification approach combining text-based features (e.g., TF-IDF, n-grams) and graph-based features derived from conversation structure (e.g., reply trees, topological measures).
• Develop an open-source benchmarking platform that standardizes data splits, evaluation metrics (Precision, Recall, F1), and model integration for consistent comparison.
• Leverage the Google Perspective API to generate baseline toxicity scores, using equal error rate thresholds (toxicity >0.64, severe_toxicity >0.92) for binary classification.
• Evaluate methods on three distinct abuse types: Personal Attack, Aggression, and Toxicity, using standardized train/dev/test splits provided with the corpus.

Experimental results

Research questions

• RQ1 Can a large-scale, publicly available corpus of full Wikipedia conversation threads be reconstructed from existing Wikipedia comment data and revision logs with high-quality annotations?
• RQ2 How does the inclusion of conversational context improve the performance of automated abuse detection models compared to isolated message classification?
• RQ3 To what extent do pre-trained models like Google Perspective API generalize across different abuse types (Personal Attack, Aggression, Toxicity) when applied to structured conversation data?
• RQ4 Can a hybrid model combining textual and structural features of conversations outperform purely text-based or structure-only models in abuse detection?
• RQ5 How effective is a shared benchmarking platform in enabling fair and reproducible evaluation of diverse abuse detection methods on a standardized corpus?

Key findings

• The WAC corpus contains 383,000 annotated messages across 193,000 conversation threads, with high-quality human-labeled abuse annotations, making it one of the largest publicly available conversation-level abuse detection datasets.
• The Google Perspective API achieves an F1-score of approximately 75% on the Toxicity dataset, but performance drops significantly on Personal Attack and Aggression, indicating limited generalization across abuse types.
• The severe_toxicity score from the Perspective API yields a high recall (~70%) but poor precision (~54%), suggesting a high rate of false positives, which confirms the lower reliability of automated annotations compared to human-annotated labels.
• The proposed hybrid text- and graph-based classifier achieves an F1-score of around 75% across all three abuse types, outperforming the Perspective API’s toxicity score but underperforming compared to prior work on different corpora.
• The benchmarking platform enables consistent evaluation and comparison of methods, with standardized data splits and evaluation metrics, reducing the risk of non-replicable results and promoting reproducibility in abuse detection research.
• The study highlights that graph-based features designed for linear chat conversations are less effective on Wikipedia talk pages, which often feature non-linear, multi-branch discussion structures, indicating a need for tailored structural modeling techniques.