[Paper Review] Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancement

Hui Liu, Wenya Wang+1 more|arXiv (Cornell University)|Oct 7, 2022

Sentiment Analysis and Opinion Mining4 citations

TL;DR

This paper proposes a hierarchical multi-modal sarcasm detection framework that models both atomic-level congruity (token-image patch alignment) and composition-level congruity (phrase-object relation alignment) using multi-head cross attention and graph neural networks. It further enhances performance by integrating external knowledge, particularly image captions generated via CLIP and GPT-2, achieving state-of-the-art results on a public Twitter-based sarcasm detection benchmark.

ABSTRACT

Sarcasm is a linguistic phenomenon indicating a discrepancy between literal meanings and implied intentions. Due to its sophisticated nature, it is usually challenging to be detected from the text itself. As a result, multi-modal sarcasm detection has received more attention in both academia and industries. However, most existing techniques only modeled the atomic-level inconsistencies between the text input and its accompanying image, ignoring more complex compositions for both modalities. Moreover, they neglected the rich information contained in external knowledge, e.g., image captions. In this paper, we propose a novel hierarchical framework for sarcasm detection by exploring both the atomic-level congruity based on multi-head cross attention mechanism and the composition-level congruity based on graph neural networks, where a post with low congruity can be identified as sarcasm. In addition, we exploit the effect of various knowledge resources for sarcasm detection. Evaluation results on a public multi-modal sarcasm detection dataset based on Twitter demonstrate the superiority of our proposed model.

Motivation & Objective

• To address the limitations of existing multi-modal sarcasm detection methods that focus only on atomic-level token-patch alignment and ignore compositional semantic mismatches.
• To improve sarcasm detection by modeling congruity at both atomic and composition levels using graph-based structures for text and visual modalities.
• To investigate the effectiveness of external knowledge, particularly image captions, in enhancing sarcasm detection beyond raw image and text inputs.
• To develop a unified framework that jointly leverages multi-granularity congruity and knowledge-enhanced representations for more robust sarcasm identification.

Proposed method

• Uses multi-head cross attention to compute similarity scores between individual text tokens and image patches, modeling atomic-level congruity.
• Constructs textual dependency graphs and visual spatial relation graphs to capture compositional semantics in both modalities.
• Applies graph attention networks (GATs) to learn contextualized representations at the composition level, enabling modeling of phrase-object relations.
• Integrates external knowledge by generating descriptive captions from images using CLIP and GPT-2, which are then used as auxiliary signals for congruity modeling.
• Concatenates atomic-level and composition-level congruity features to form a unified representation for sarcasm classification.
• Employs a hierarchical architecture that refines low-level congruities through higher-level graph-based reasoning to detect complex semantic inconsistencies.

Experimental results

Research questions

• RQ1 Can modeling both atomic-level and composition-level congruity improve multi-modal sarcasm detection beyond single-granularity approaches?
• RQ2 How effective is external knowledge, particularly image captions, in enhancing sarcasm detection when visual and textual inputs are semantically inconsistent?
• RQ3 Does the integration of pre-trained vision-language models (e.g., CLIP) and language models (e.g., GPT-2) improve the interpretability and robustness of sarcasm detection?
• RQ4 To what extent do graph-based modeling of semantic and spatial dependencies enhance the detection of complex sarcasm patterns?

Key findings

• The proposed model achieves state-of-the-art performance on the public Twitter-based multi-modal sarcasm detection dataset, outperforming existing methods.
• The integration of image captions generated via CLIP and GPT-2 significantly improves detection accuracy, especially in cases with subtle or figurative sarcasm.
• Visualization of congruity scores confirms that the composition-level module effectively down-weights mismatched regions (e.g., furniture in a storm image), refining detection beyond atomic attention.
• The hierarchical modeling approach successfully identifies complex inconsistencies, such as contradictions between a phrase and a group of image patches, which single-granularity models miss.
• The model demonstrates robustness in detecting sarcasm even when the image contains ambiguous or low-level visual cues, thanks to knowledge-enhanced representations.
• Ablation studies show that knowledge enhancement via image captions provides a consistent performance gain, particularly in cases where visual features alone are insufficient.