[Paper Review] BLUE at Memotion 2.0 2022: You have my Image, my Text and my Transformer

Ana-Maria Bucur, Adrian Cosma+1 more|arXiv (Cornell University)|Feb 15, 2022

Sentiment Analysis and Opinion Mining7 citations

TL;DR

This paper presents team BLUE’s winning approach to the MEMOTION 2.0 2022 shared task, proposing a text-only BERT-based method and a Multi-Modal-Multi-Task (MMT) transformer that fuses image and text features. The MMT model, enhanced with CLIP and CORAL for ordinal regression, achieved first place in sentiment analysis (F1=0.5318), second in humour detection (F1=0.8059), and third in sarcasm/offensiveness tasks, with the highest average score across all tasks.

ABSTRACT

Memes are prevalent on the internet and continue to grow and evolve alongside our culture. An automatic understanding of memes propagating on the internet can shed light on the general sentiment and cultural attitudes of people. In this work, we present team BLUE's solution for the second edition of the MEMOTION shared task. We showcase two approaches for meme classification (i.e. sentiment, humour, offensive, sarcasm and motivation levels) using a text-only method using BERT, and a Multi-Modal-Multi-Task transformer network that operates on both the meme image and its caption to output the final scores. In both approaches, we leverage state-of-the-art pretrained models for text (BERT, Sentence Transformer) and image processing (EfficientNetV4, CLIP). Through our efforts, we obtain first place in task A, second place in task B and third place in task C. In addition, our team obtained the highest average score for all three tasks.

Motivation & Objective

• To develop robust multimodal models for fine-grained meme classification across sentiment, humour, sarcasm, offensiveness, and motivation.
• To investigate the effectiveness of combining textual and visual features using state-of-the-art pretrained models in multimodal meme understanding.
• To evaluate the impact of different modality combinations and feature extractors (e.g., CLIP, EfficientNetV4) on classification performance.
• To address the challenge of ordinal regression in meme emotion intensity prediction using CORAL.
• To improve generalization on imbalanced meme datasets through self-supervised pretraining and lexical normalization.

Proposed method

• Fine-tuned BERT and Sentence Transformers on textual captions extracted from memes for text-only classification.
• Designed a Multi-Modal-Multi-Task (MMT) transformer that jointly processes image and text embeddings from EfficientNetV4 and BERT.
• Integrated CLIP features as an additional visual encoder to improve multimodal representation learning.
• Applied CORAL (Domain Adaptation) for ordinal regression to model intensity levels of emotions like humour and sarcasm.
• Conducted ablation studies across modality combinations (text-only, image-only, CLIP-only, and fused) to evaluate feature contributions.
• Used weighted F1-score as the primary metric across all tasks, with model ensembling and hyperparameter tuning for final submissions.

Experimental results

Research questions

• RQ1 How does a text-only BERT-based approach compare to multimodal models in classifying meme sentiment and emotions?
• RQ2 What is the relative contribution of image, text, and CLIP features to performance across different meme classification tasks?
• RQ3 Can CORAL-based domain adaptation improve ordinal regression performance for intensity-level emotion prediction in memes?
• RQ4 Is there a single optimal modality combination that generalizes across all meme emotion tasks, or does performance vary per task?
• RQ5 How do self-supervised pretraining and lexical normalization improve robustness on noisy, imbalanced meme datasets?

Key findings

• The MMT transformer with fused image, text, and CLIP features achieved the highest average F1 score (0.6273) across all three tasks, outperforming all other teams.
• The text-only BERT model achieved second place in humour detection (F1=0.7743) and performed best on humour and offensiveness tasks.
• The MMT model secured first place in sentiment analysis (F1=0.5318), outperforming the text-only model (F1=0.5072) on this task.
• For sarcasm and offensive content, the MMT model significantly outperformed the text-only model, achieving F1 scores of 0.8191 and 0.5581 respectively.
• Ablation studies showed minimal performance differences across modality combinations, indicating that CLIP and image features provided consistent, small gains.
• The team achieved the highest average score across all three tasks, demonstrating the robustness and generalization of the MMT approach.