[Paper Review] A Study of Question Effectiveness Using Reddit "Ask Me Anything" Threads

Kristjan Arumae, Guo-Jun Qi+1 more|arXiv (Cornell University)|May 25, 2018

Expert finding and Q&A systems19 references3 citations

TL;DR

This paper proposes a context-aware convolutional neural network (CNN) to predict which questions in Reddit 'Ask Me Anything' (AMA) threads are answered by hosts, using a large-scale dataset of 400,000 questions. The model achieves a 53.7% F1 score on a 10k validation set, outperforming baselines and showing strong alignment with human judgments on question effectiveness.

ABSTRACT

Asking effective questions is a powerful social skill. In this paper we seek to build computational models that learn to discriminate effective questions from ineffective ones. Armed with such a capability, future advanced systems can evaluate the quality of questions and provide suggestions for effective question wording. We create a large-scale, real-world dataset that contains over 400,000 questions collected from Reddit "Ask Me Anything" threads. Each thread resembles an online press conference where questions compete with each other for attention from the host. This dataset enables the development of a class of computational models for predicting whether a question will be answered. We develop a new convolutional neural network architecture with variable-length context and demonstrate the efficacy of the model by comparing it with state-of-the-art baselines and human judges.

Motivation & Objective

• To develop computational models that can automatically assess question effectiveness in real-world online Q&A settings.
• To investigate whether question wording and context influence whether a question is answered by an AMA host.
• To create a large-scale, real-world dataset for studying question answerability in community-driven Q&A forums.
• To compare machine predictions with human judgments and AMA host behavior to evaluate model performance and alignment.
• To explore the factors that make certain questions more likely to be answered, including semantic, syntactic, and topical features.

Proposed method

• Constructed a large-scale dataset of 400,000 questions from Reddit AMA threads, including thread titles, host introductions, and user questions.
• Designed a novel context-aware CNN architecture that processes variable-length context and combines multiple n-gram filter windows (1–5-grams) with varying filter counts.
• Trained the model using a combination of unigram, bigram, trigram, and higher-order n-gram features to capture both local and distributed semantic patterns.
• Evaluated model performance using a 10k validation set and compared against state-of-the-art baselines and human annotators.
• Conducted a human evaluation study via Amazon Mechanical Turk, where 5 annotators labeled 300 question pairs from the same AMA thread to assess preference and consensus.
• Used majority voting from human annotators as a proxy for human judgment and compared model predictions against both human and host behavior.

Experimental results

Research questions

• RQ1 What features of a question make it more likely to be answered by an AMA host?
• RQ2 Can a deep learning model effectively predict which questions will be answered based on their wording and context?
• RQ3 How well do human judgments on question effectiveness align with actual AMA host choices?
• RQ4 To what extent does the model’s prediction performance match human consensus versus host behavior?
• RQ5 How does training data size affect the generalization and stability of the question answerability prediction model?

Key findings

• The context-aware CNN model achieved a 53.7% F1 score on the 10k validation set, outperforming individual baseline CNN models and state-of-the-art methods.
• The model performed best with 300 filters and a 1-word window, indicating strong reliance on unigram features for prediction.
• Human annotators reached consensus on 63% of question pairs (4 or more out of 5 agreeing), with 30% achieving full agreement among all 5 annotators.
• Human annotators agreed with AMA hosts on 55% of question pairs, while the model showed slightly higher agreement with human judgments (54.33%) than with host choices (49.33%).
• An undergraduate student achieved 68.6% agreement with human annotators, outperforming the model’s alignment with host behavior (49.33%).
• Model performance improved steadily with increased training data size, peaking at 53.7% F1 with 100k training samples.