[Paper Review] Stance Classification for Rumour Analysis in Twitter: Exploiting Affective Information and Conversation Structure

Endang Wahyu Pamungkas, Valerio Basile+1 more|arXiv (Cornell University)|Jan 7, 2019

Misinformation and Its Impacts8 references18 citations

TL;DR

This paper proposes a stance classification model for rumour analysis on Twitter that integrates conversation structure and affective features to improve detection of user stances—supporting, denying, querying, or commenting—on emerging rumours. The approach outperforms prior state-of-the-art systems on the SemEval-2017 Task 8 benchmark without relying on deep learning, demonstrating the effectiveness of structured and affective feature engineering in stance detection for misinformation mitigation.

ABSTRACT

Analysing how people react to rumours associated with news in social media is an important task to prevent the spreading of misinformation, which is nowadays widely recognized as a dangerous tendency. In social media conversations, users show different stances and attitudes towards rumourous stories. Some users take a definite stance, supporting or denying the rumour at issue, while others just comment it, or ask for additional evidence related to the veracity of the rumour. On this line, a new shared task has been proposed at SemEval-2017 (Task 8, SubTask A), which is focused on rumour stance classification in English tweets. The goal is predicting user stance towards emerging rumours in Twitter, in terms of supporting, denying, querying, or commenting the original rumour, looking at the conversation threads originated by the rumour. This paper describes a new approach to this task, where the use of conversation-based and affective-based features, covering different facets of affect, has been explored. Our classification model outperforms the best-performing systems for stance classification at SemEval-2017 Task 8, showing the effectiveness of the feature set proposed.

Motivation & Objective

• To address the challenge of detecting user stances toward emerging rumours in Twitter conversations, particularly in the context of misinformation spread.
• To improve stance classification accuracy by incorporating both conversation structure and affective information beyond basic sentiment.
• To overcome limitations of existing systems, especially in handling rare classes like 'deny' and ambiguous cases such as sarcasm or mixed intent.
• To provide a robust, non-deep-learning-based model that leverages hand-crafted features for interpretability and performance.
• To analyze failure cases and identify persistent challenges such as sarcasm, literal citations, and class imbalance in annotation.

Proposed method

• The method constructs a feature matrix combining conversation-based features (e.g., reply depth, thread structure, reply directionality) and affective features from diverse sentiment and affect lexicons (e.g., LIWC, EmoLex, SentiStrength).
• Dialogue act features are incorporated to model communicative intent, linking affective content to functions like agreement, denial, or information request.
• The model uses a supervised learning approach (e.g., SVM or logistic regression) trained on the SemEval-2017 Task 8 dataset, with features engineered to reflect both linguistic and structural context.
• A feature importance analysis is conducted to evaluate contributions of different feature groups, including structural, affective, and dialogue-act components.
• Error analysis is performed on misclassified cases, focusing on sarcasm, literal citations, and borderline annotations to identify model limitations.
• Class imbalance is addressed through resampling techniques, showing improved performance on the 'deny' class when balanced.

Experimental results

Research questions

• RQ1 How do conversation structure features contribute to improving stance classification performance in rumour threads?
• RQ2 To what extent do affective features capturing diverse affective dimensions (e.g., fear, surprise, trust) enhance stance detection beyond basic sentiment?
• RQ3 Can a system based on hand-crafted features outperform deep learning models in stance classification for rumour analysis?
• RQ4 What are the main failure modes of current stance classification systems, particularly regarding sarcasm, citations, and ambiguous annotations?
• RQ5 How does class imbalance impact model performance, and can it be mitigated effectively with feature engineering and data balancing?

Key findings

• The proposed model achieves state-of-the-art performance on the SemEval-2017 Task 8 benchmark, outperforming all previously published systems for rumour stance classification.
• The inclusion of conversation-based and affective features significantly improves classification accuracy, with affective features contributing notably to distinguishing between 'support' and 'comment' stances.
• The model struggles with sarcasm, misclassifying sarcastic supportive replies as 'comments' due to implicit tone, highlighting a key limitation in affective feature design.
• Literal citations of source tweets are frequently misclassified as 'comments' when the original tweet is supportive, suggesting a need for pre-processing to detect quoted content.
• The 'deny' class is under-predicted in the original dataset due to class imbalance, but performance improves substantially when the dataset is balanced, indicating that data distribution heavily influences model outcomes.
• Mixed or borderline cases in the gold standard annotation—such as replies that both question and comment—contribute to misclassification and reveal annotation challenges.