[Paper Review] Automatic Identification of Sarcasm Target: An Introductory Approach

Aditya Joshi, Pranav Goel+2 more|arXiv (Cornell University)|Oct 22, 2016

Sentiment Analysis and Opinion Mining21 references3 citations

TL;DR

This paper introduces sarcasm target identification, a novel task to extract the entity or concept ridiculed in a sarcastic utterance. It proposes a hybrid approach combining rule-based and statistical classifiers, which outperforms two baselines and individual components on book snippets and tweets, achieving a Dice score of 39.63 with the hybrid OR configuration.

ABSTRACT

Past work in computational sarcasm deals primarily with sarcasm detection. In this paper, we introduce a novel, related problem: sarcasm target identification i.e., extracting the target of ridicule in a sarcastic sentence). We present an introductory approach for sarcasm target identification. Our approach employs two types of extractors: one based on rules, and another consisting of a statistical classifier. To compare our approach, we use two baselines: a naïve baseline and another baseline based on work in sentiment target identification. We perform our experiments on book snippets and tweets, and show that our hybrid approach performs better than the two baselines and also, in comparison with using the two extractors individually. Our introductory approach establishes the viability of sarcasm target identification, and will serve as a baseline for future work.

Motivation & Objective

• To address the gap in computational sarcasm research by identifying the target of ridicule in sarcastic texts, a task previously unexplored.
• To develop a method that accurately extracts the specific entity or concept being ridiculed in a sarcastic sentence.
• To evaluate the performance of a hybrid approach combining rule-based and statistical extraction techniques on real-world sarcastic text.
• To establish a baseline for future research in sarcasm target identification using manually annotated datasets.
• To analyze challenges such as multiple targets and cases where the target is not explicitly present in the text (labeled 'Outside').

Proposed method

• The approach uses a rule-based extractor implementing nine linguistic rules to identify sarcasm targets based on syntactic and lexical patterns.
• A statistical classifier processes each word in a sentence as a separate instance, using part-of-speech (POS) and sentiment features to predict if the word is a sarcasm target.
• Outputs from the two extractors are combined using two configurations: OR (union of predictions) and AND (intersection of predictions).
• The system is trained and evaluated on two manually annotated datasets: one of book snippets and one of tweets.
• Performance is measured using exact match (EM), F1, and Dice scores, with special attention to the 'Outside' case where the target is not in the text.
• Error analysis focuses on cases where the target is not present in the sentence, identifying misclassification patterns.

Experimental results

Research questions

• RQ1 Can a hybrid approach combining rule-based and statistical methods effectively identify the target of ridicule in sarcastic texts?
• RQ2 How does the performance of the proposed method compare to a simple baseline and a sentiment target identification baseline?
• RQ3 What challenges arise when the sarcasm target is not explicitly present in the text (i.e., the 'Outside' case)?
• RQ4 How do the OR and AND fusion strategies compare in terms of precision, recall, and F1 score?
• RQ5 What are the main error patterns in the 'Outside' case, and how can they be mitigated in future work?

Key findings

• The hybrid OR approach achieved the highest performance with a Dice score of 39.63 on the combined dataset, outperforming both baselines and individual extractors.
• The rule-based and statistical extractors individually performed worse than the hybrid OR configuration, indicating complementary strengths.
• Performance on the 'Outside' case showed a significant drop in Dice score (from 39.63 to 20.45), though exact match remained relatively stable, indicating partial matching issues.
• The system correctly identified 'Outside' for sentences like 'Yeah, just ignore me. That is TOTALLY the right way to handle this!', but incorrectly assigned targets like 'I' or 'me' in similar cases.
• Error analysis revealed that misclassification in 'Outside' cases often stemmed from over-interpretation of pronouns or implied referents not present in the text.
• The study establishes the first baseline for sarcasm target identification, with publicly available annotated datasets for future research.