[Paper Review] Transformer-based Context-aware Sarcasm Detection in Conversation Threads from Social Media

Xiangjue Dong, Changmao Li+1 more|arXiv (Cornell University)|May 22, 2020

Sentiment Analysis and Opinion Mining15 references4 citations

TL;DR

This paper proposes a transformer-based context-aware sarcasm detection model that leverages multi-head attention across the target utterance and its full conversation thread to improve sarcasm prediction. By incorporating contextual discourse features, the model achieves state-of-the-art F1 scores of 79.0% on Twitter and 75.0% on Reddit, outperforming target-only baselines by 3.1% and 7.0% respectively.

ABSTRACT

We present a transformer-based sarcasm detection model that accounts for the context from the entire conversation thread for more robust predictions. Our model uses deep transformer layers to perform multi-head attentions among the target utterance and the relevant context in the thread. The context-aware models are evaluated on two datasets from social media, Twitter and Reddit, and show 3.1% and 7.0% improvements over their baselines. Our best models give the F1-scores of 79.0% and 75.0% for the Twitter and Reddit datasets respectively, becoming one of the highest performing systems among 36 participants in this shared task.

Motivation & Objective

• To address the challenge of sarcasm detection in social media, where surface-level positivity masks negative sentiment.
• To overcome the limitations of target-utterance-only models that fail to capture contextual nuance critical for sarcasm detection.
• To evaluate the impact of full conversation thread context on sarcasm detection performance using state-of-the-art transformer architectures.
• To demonstrate that context-aware modeling significantly improves detection accuracy, especially in long and complex threads.
• To establish a strong baseline and publicly release resources for future research in sarcasm detection.

Proposed method

• The model uses a transformer encoder to jointly encode the target utterance and its surrounding conversation context using multi-head self-attention mechanisms.
• Contextual representations are learned by attending across all utterances in the thread, enabling the model to capture discourse-level incongruities.
• Three pre-trained transformer models—RoBERTa-Large, BERT-Large, and ALBERT-xxLarge—are fine-tuned for the sarcasm detection task.
• The input is structured as a sequence of [CLS] + context utterances + [SEP] + target utterance + [SEP], preserving sequential order.
• Model training employs binary cross-entropy loss with sigmoid activation for multi-label classification of sarcasm vs. non-sarcasm.
• The model is evaluated on two social media datasets: Twitter (using #sarcasm hashtags) and Reddit (from the Self-Annotated Reddit Corpus).

Experimental results

Research questions

• RQ1 Can incorporating the full conversation thread significantly improve sarcasm detection performance compared to models using only the target utterance?
• RQ2 How do different pre-trained transformer architectures (RoBERTa, BERT, ALBERT) perform when extended to context-aware sarcasm detection?
• RQ3 Does context awareness help detect subtle sarcasm that is ambiguous in isolation but becomes clear with surrounding discourse?
• RQ4 How does model performance vary across datasets with differing thread and utterance length distributions, such as Reddit’s longer test set?
• RQ5 What types of errors are introduced or mitigated when using context-aware modeling versus target-oriented models?

Key findings

• The context-aware model achieves a 7.0% absolute improvement in F1 score on the Reddit dataset compared to the target-oriented baseline, with a final F1 of 75.0%.
• On the Twitter dataset, the context-aware model improves F1 by 3.1% over the baseline, reaching 79.0% F1, ranking second in the shared task.
• Error analysis reveals that context-aware models correctly classify 100% of cases where target-only models fail due to lack of contextual cues.
• In cases where context-aware models make errors, the issue often stems from noisy or overly long context, suggesting a trade-off between context richness and model robustness.
• The model outperforms 36 other participants in the shared task, establishing it as one of the highest-performing systems.
• The model's performance gain is most pronounced on Reddit, where test set conversation threads are significantly longer than in the training set, highlighting the value of context in long-form discourse.