[Paper Review] Getting Reliable Annotations for Sarcasm in Online Dialogues

Reid Swanson, Stephanie M. Lukin+3 more|arXiv (Cornell University)|Sep 4, 2017

Mobile Crowdsensing and Crowdsourcing15 references3 citations

TL;DR

This paper investigates reliable sarcasm annotation in online dialogues using crowdsourced labeling on Amazon Mechanical Turk. It compares statistical reliability measures—Kappa, Karger’s, Majority Vote, and EM—and finds that simple majority voting achieves high reliability with only 3–7 annotators per utterance, resolving ambiguity in most cases and requiring up to 25 annotations for the most ambiguous instances. The key contribution is a practical, scalable method for obtaining high-quality sarcasm annotations with minimal overhead.

ABSTRACT

The language used in online forums differs in many ways from that of traditional language resources such as news. One difference is the use and frequency of nonliteral, subjective dialogue acts such as sarcasm. Whether the aim is to develop a theory of sarcasm in dialogue, or engineer automatic methods for reliably detecting sarcasm, a major challenge is simply the difficulty of getting enough reliably labelled examples. In this paper we describe our work on methods for achieving highly reliable sarcasm annotations from untrained annotators on Mechanical Turk. We explore the use of a number of common statistical reliability measures, such as Kappa, Karger's, Majority Class, and EM. We show that more sophisticated measures do not appear to yield better results for our data than simple measures such as assuming that the correct label is the one that a majority of Turkers apply.

Motivation & Objective

• To address the challenge of obtaining reliably labeled sarcasm data in online dialogues, where traditional NLP annotation methods often fail due to subjectivity and nuance.
• To evaluate whether sophisticated statistical reliability measures (e.g., Karger’s, EM) outperform simple majority voting in labeling sarcasm on crowdsourced data.
• To determine the optimal number of annotators needed to achieve stable, reliable sarcasm annotations for online dialogues.
• To assess whether sarcasm annotation is inherently more ambiguous than other subjective NLP tasks, requiring more annotators.
• To provide a scalable, empirically validated method for generating reliable sarcasm datasets from untrained annotators.

Proposed method

• The authors collected 25 annotations per utterance from untrained Mechanical Turk workers for 600 online dialogue responses, using a gold-standard set of 100 sarcasm-labeled examples.
• They applied multiple reliability estimation methods: Cohen’s Kappa, Karger’s algorithm, Majority Vote, and Expectation-Maximization (EM) to infer ground truth from noisy labels.
• The study used a stratified sampling approach to identify the most ambiguous cases and tracked label convergence across increasing numbers of annotators (from 3 to 25).
• Label stability was measured by tracking how often items changed their classification (sarcasm vs. non-sarcasm) as more annotations were added.
• The final gold standard was established using Karger’s algorithm with full annotation sets, and results were compared across methods to assess accuracy and convergence.
• The authors analyzed label switching behavior in highly ambiguous cases, focusing on how initial ambiguity (ratio near 0.5) evolved with increasing annotation counts.

Experimental results

Research questions

• RQ1 Does using sophisticated reliability measures like Karger’s or EM yield better sarcasm annotation results than simple majority voting?
• RQ2 What is the minimum number of annotators required to achieve stable and reliable sarcasm labels in online dialogues?
• RQ3 Are sarcasm annotations inherently more ambiguous than other subjective NLP tasks, requiring more annotators to reach consensus?
• RQ4 How do label assignments change as more annotations are collected, and when does convergence occur?
• RQ5 To what extent do initial ambiguous labels (near 0.5 ratio of sarcastic to non-sarcastic votes) stabilize with additional annotations?

Key findings

• Majority voting achieved 82.5% accuracy on the gold standard with only 3 annotators, indicating strong reliability even with minimal labeling.
• For the most ambiguous 400 utterances, accuracy increased from ~80% at 3 annotators to ~90% at 10 annotators, with minimal gains beyond 23 annotations.
• Only 9 out of 600 utterances remained ambiguous after 25 annotations, suggesting that near-perfect consensus is achievable with moderate annotation effort.
• Label convergence occurred by 7 annotations for most items, with only a small fraction of highly ambiguous cases continuing to shift labels beyond 10–15 annotations.
• The study found that 7 annotators are sufficient to stabilize sarcasm labels, supporting the conclusion that sarcasm annotation does not require more annotators than other subjective NLP tasks.
• The results show that simple majority voting performs as well as complex models like Karger’s and EM, with minimal computational cost.