[Paper Review] Chaff from the Wheat : Characterization and Modeling of Deleted Questions on Stack Overflow

Denzil Correa, Ashish Sureka|arXiv (Cornell University)|Jan 2, 2014

Expert finding and Q&A systemsComputer Science17 citations

TL;DR

This paper presents the first large-scale study of deleted questions on Stack Overflow, analyzing their characteristics and building a predictive model using 47 features across user profile, community behavior, content, and syntax. It achieves 66% accuracy in predicting deletion at time of posting, revealing insights into low-quality content patterns and the pyramidal quality structure of questions on the platform.

ABSTRACT

Stack Overflow is the most popular CQA for programmers on the web with 2.05M users, 5.1M questions and 9.4M answers. Stack Overflow has explicit, detailed guidelines on how to post questions and an ebullient moderation community. Despite these precise communications and safeguards, questions posted on Stack Overflow can be extremely off topic or very poor in quality. Such questions can be deleted from Stack Overflow at the discretion of experienced community members and moderators. We present the first study of deleted questions on Stack Overflow. We divide our study into two parts (i) Characterization of deleted questions over approx. 5 years (2008-2013) of data, (ii) Prediction of deletion at the time of question creation. Our characterization study reveals multiple insights on question deletion phenomena. We observe a significant increase in the number of deleted questions over time. We find that it takes substantial time to vote a question to be deleted but once voted, the community takes swift action. We also see that question authors delete their questions to salvage reputation points. We notice some instances of accidental deletion of good quality questions but such questions are voted back to be undeleted quickly. We discover a pyramidal structure of question quality on Stack Overflow and find that deleted questions lie at the bottom (lowest quality) of the pyramid. We also build a predictive model to detect the deletion of question at the creation time. We experiment with 47 features based on User Profile, Community Generated, Question Content and Syntactic style and report an accuracy of 66%. Our feature analysis reveals that all four categories of features are important for the prediction task. Our findings reveal important suggestions for content quality maintenance on community based question answering websites.

Motivation & Objective

• To understand the nature and causes of deleted questions on Stack Overflow despite strict guidelines and active moderation.
• To identify patterns in question deletion, including timing, voting behavior, and self-deletion by authors.
• To develop a predictive model that flags low-quality questions at creation time to improve content quality and reduce moderator burden.
• To analyze the relative importance of different feature types—user profile, community behavior, content, and syntax—for predicting deletion.

Proposed method

• Collected and analyzed 470,096 questions from Stack Overflow (2008–2013), with 235,048 deleted and 235,048 non-deleted questions.
• Categorized 47 predictive features into four groups: user profile, community-generated signals, question content, and syntactic style.
• Employed an ensemble learning approach using Adaboost with decision trees as base classifiers and SAMME.R boosting algorithm.
• Used 70-30% training-testing split with 10-fold cross-validation to prevent overfitting and ensure robust evaluation.
• Incrementally added feature sets to evaluate their contribution to prediction performance using F1 score, accuracy, and AUC.
• Used feature importance analysis from Adaboost to identify top discriminative features across all four feature categories.

Experimental results

Research questions

• RQ1 What are the temporal trends and behavioral patterns associated with the deletion of questions on Stack Overflow?
• RQ2 How do community voting and self-deletion by authors influence the deletion process, and what motivates these actions?
• RQ3 What is the structural relationship between question quality levels on Stack Overflow, and where do deleted questions fit in this hierarchy?
• RQ4 Which combination of features—user profile, community behavior, content, and syntax—best predicts whether a question will be deleted?
• RQ5 To what extent can a machine learning model predict deletion at the time of question creation, and how do individual feature sets contribute to this?

Key findings

• The number of deleted questions on Stack Overflow has shown a significant increasing trend over the past five years, particularly in the last two years.
• Although community members take substantial time to vote for deletion, the actual deletion process is swift once a consensus is reached.
• Authors frequently delete their own questions to recover lost reputation points, indicating a reputational incentive behind self-deletion.
• Accidental deletion of high-quality questions occurs but is quickly reversed through community voting to undelete.
• A pyramidal structure of question quality exists on Stack Overflow, with deleted questions occupying the lowest tier (worst quality), followed by closed and then active questions.
• The predictive model achieves 66% accuracy in identifying deleted questions at creation time, with incremental improvement as each feature set is added, and all four feature categories contribute significantly to prediction performance.