[Paper Review] Overview of the Wikidata Vandalism Detection Task at WSDM Cup 2017

Stefan Heindorf, Martin Potthast+2 more|arXiv (Cornell University)|Dec 16, 2017

Wikis in Education and Collaboration18 references3 citations

TL;DR

This paper presents the Wikidata Vandalism Detection Task at WSDM Cup 2017, introducing a real-time, online learning framework for detecting malicious edits in Wikidata. The best-performing model achieved an AUC of 0.947 on the ROC curve and 0.458 on the PR curve, with all submissions and evaluation code released open source to enable reproducibility and future research.

ABSTRACT

We report on the Wikidata vandalism detection task at the WSDM Cup 2017. The task received five submissions for which this paper describes their evaluation and a comparison to state of the art baselines. Unlike previous work, we recast Wikidata vandalism detection as an online learning problem, requiring participant software to predict vandalism in near real-time. The best-performing approach achieves a ROC-AUC of 0.947 at a PR-AUC of 0.458. In particular, this task was organized as a software submission task: to maximize reproducibility as well as to foster future research and development on this task, the participants were asked to submit their working software to the TIRA experimentation platform along with the source code for open source release.

Motivation & Objective

• To develop a scalable, real-time system for detecting vandalism in Wikidata, a rapidly growing knowledge base with millions of monthly edits.
• To address the challenge of detecting malicious edits that could compromise downstream systems like search engines and question-answering tools.
• To foster collaborative research by organizing a shared task with standardized evaluation and open-source code release.
• To evaluate models under both semi-automatic (ranking for review) and fully automatic (reversion decision) detection scenarios.
• To analyze the performance limits of current feature sets and identify opportunities for future innovation in feature engineering.

Proposed method

• The task was structured as a software submission challenge, requiring participants to deploy their models on the TIRA platform for remote execution.
• A new, updated Wikidata Vandalism Corpus 2016 was compiled from revision history, containing labeled edits for training and evaluation.
• Participants were required to implement a client-server architecture to process revisions in near real-time, simulating production deployment.
• Evaluation was based on ROC AUC and PR AUC, with models ranked on their ability to detect vandalism across diverse precision-recall trade-offs.
• A meta-classifier was trained on the ensemble of all submitted models to assess the potential of model stacking and feature set limitations.
• All evaluation frameworks and most participant codebases were released as open source to ensure reproducibility and community reuse.

Experimental results

Research questions

• RQ1 What is the performance of machine learning models in detecting vandalism in Wikidata under real-time, online learning constraints?
• RQ2 How do different feature sets and model architectures compare in detecting vandalism across varying precision-recall trade-offs?
• RQ3 Can ensemble methods improve detection performance beyond individual models, and what are the limiting factors?
• RQ4 What are the main barriers to participation in large-scale, real-time shared tasks involving complex software deployment?
• RQ5 To what extent do current feature sets constrain detection performance, and what alternative features could improve future models?

Key findings

• The best-performing model achieved a ROC AUC of 0.947 and a PR AUC of 0.458, demonstrating strong detection capability under real-time constraints.
• The WDVD baseline model outperformed all participant submissions in the fully automatic detection scenario, indicating strong robustness.
• Ensemble modeling with all submitted approaches yielded only a marginal performance improvement, suggesting that feature diversity is limited.
• Only five teams successfully submitted working software, highlighting challenges in deployment and system complexity despite open-source incentives.
• The high barrier to entry was attributed to the large dataset size, real-time processing requirements, and remote virtual machine deployment on TIRA.
• The study concluded that future progress depends on developing fundamentally new feature sets, such as those based on knowledge graph embeddings or psychological user modeling.