[Paper Review] Ensemble Models for Detecting Wikidata Vandalism with Stacking - Team Honeyberry Vandalism Detector at WSDM Cup 2017

Tomoya Yamazaki, Mei Sasaki+3 more|arXiv (Cornell University)|Dec 19, 2017

Software Engineering Research2 references3 citations

TL;DR

This paper presents an ensemble stacking approach for detecting Wikidata vandalism in the WSDM Cup 2017 competition. By combining under-sampling, feature selection, and stacked meta-learner models, the method achieves an AUC-ROC of 0.94412, demonstrating significant improvements over baseline models through systematic validation of each technique's impact.

ABSTRACT

The WSDM Cup 2017 is a binary classification task for classifying Wikidata revisions into vandalism and non-vandalism. This paper describes our method using some machine learning techniques such as under-sampling, feature selection, stacking and ensembles of models. We confirm the validity of each technique by calculating AUC-ROC of models using such techniques and not using them. Additionally, we analyze the results and gain useful insights into improving models for the vandalism detection task. The AUC-ROC of our final submission after the deadline resulted in 0.94412.

Motivation & Objective

• To develop a robust binary classification system for identifying vandalism in Wikidata revisions.
• To improve detection performance on an imbalanced dataset where vandalism instances are rare.
• To evaluate the effectiveness of individual preprocessing and modeling techniques in isolation and in combination.
• To optimize model performance through ensemble stacking of diverse base learners.
• To provide actionable insights into feature engineering and model selection for vandalism detection tasks.

Proposed method

• The authors apply under-sampling to balance the highly imbalanced training data, reducing false positives from non-vandalism revisions.
• They perform feature selection to retain only the most discriminative features, improving model generalization and reducing overfitting.
• Multiple base models (e.g., XGBoost, Logistic Regression) are trained independently on the preprocessed data to form a diverse ensemble.
• A meta-learner model is trained on the predictions of the base models to learn optimal weighting and stacking strategy.
• The final stacked ensemble combines predictions from all base models using the meta-learner to produce the final classification output.
• The entire pipeline is validated using AUC-ROC to measure performance and assess the contribution of each technique.

Experimental results

Research questions

• RQ1 How does under-sampling affect the AUC-ROC performance of vandalism detection models?
• RQ2 To what extent does feature selection improve model generalization and reduce overfitting?
• RQ3 Can stacking multiple diverse base models significantly outperform individual models?
• RQ4 What is the relative contribution of each component (under-sampling, feature selection, stacking) to the final model performance?
• RQ5 What insights can be gained from analyzing model behavior and feature importance in the context of vandalism detection?

Key findings

• The final stacked ensemble model achieved an AUC-ROC of 0.94412, representing the best performance in the WSDM Cup 2017 competition.
• Under-sampling significantly improved model performance by reducing the bias toward non-vandalism classes.
• Feature selection contributed to better generalization and reduced overfitting, especially in high-dimensional feature spaces.
• Stacking of diverse base models outperformed individual models, demonstrating the effectiveness of meta-learning in combining predictions.
• The ablation study confirmed that each component—under-sampling, feature selection, and stacking—contributed positively to the final AUC-ROC.
• The analysis revealed that certain structural and edit pattern features were highly predictive of vandalism, offering insights for future feature engineering.