[Paper Review] Memes-as-Replies: Can Models Select Humorous Manga Panel Responses?

Ryosuke Kohita, Seiichiro Yoshioka|arXiv (Cornell University)|Jan 21, 2026

Humor Studies and Applications0 citations

TL;DR

The paper defines Meme Reply Selection and introduces MaMe-Re, a large benchmark of 100,000 context-meme pairs using Japanese manga panels to study humor in interactive meme replies, evaluating various similarity-based and preference-based modeling approaches. It finds that preference-based LLMs can capture complex social cues like irony and exaggeration, but visual information often does not help, and distinguishing subtleties of wit remains challenging.

ABSTRACT

Memes are a popular element of modern web communication, used not only as static artifacts but also as interactive replies within conversations. While computational research has focused on analyzing the intrinsic properties of memes, the dynamic and contextual use of memes to create humor remains an understudied area of web science. To address this gap, we introduce the Meme Reply Selection task and present MaMe-Re (Manga Meme Reply Benchmark), a benchmark of 100,000 human-annotated pairs (500,000 total annotations from 2,325 unique annotators) consisting of openly licensed Japanese manga panels and social media posts. Our analysis reveals three key insights: (1) large language models (LLMs) show preliminary evidence of capturing complex social cues such as exaggeration, moving beyond surface-level semantic matching; (2) the inclusion of visual information does not improve performance, revealing a gap between understanding visual content and effectively using it for contextual humor; (3) while LLMs can match human judgments in controlled settings, they struggle to distinguish subtle differences in wit among semantically similar candidates. These findings suggest that selecting contextually humorous replies remains an open challenge for current models.

Motivation & Objective

• Formalize Meme Reply Selection as a task focusing on humor emerging from meme-context interactions.
• Provide MaMe-Re, an openly licensed benchmark with 100,000 context-meme pairs and 500,000 annotations.
• Analyze how current models perceive and leverage multimodal cues to select humorous manga panel replies.

Proposed method

• Define a scoring function s(c, m) that rates funniness of meme m given context c.
• Create MaMe-Re with 250 contexts, 400 manga panels, and 100,000 context-meme pairs annotated by five workers each.
• Evaluate two main strategies: similarity-based selection (cosine similarity on embeddings) and preference-based selection (LLM-based funniest selection).
• Compare text-only, multimodal, and vision-language embeddings and models across multiple families (Text-Emb, LLM-Emb, Multi-Emb, VLM-Emb).
• Use Score@1, Consensus Hit Rate (CHR), and nDCG@5 as evaluation metrics; explore recoverability via retrieve-and-rerank and controlled experiments.

Figure 1: Overview of memes-as-replies. (a) Example of meme use on SNS. (b) Visualization of the Meme Reply Selection task. (c) MaMe-Re benchmark with crowdsourced humor labels.

Experimental results

Research questions

• RQ1 Can LLMs capture complex humor cues like irony or exaggeration in meme replies?
• RQ2 Does adding visual information to textual context improve humor-specific meme selection?
• RQ3 Are preference-based (LLM) approaches consistently better than similarity-based embeddings for humor in meme replies?
• RQ4 Do models struggle more when choosing from semantically similar candidates, and why?

Key findings

• LLMs show preliminary evidence of modeling complex social cues in humor, particularly irony and exaggeration.
• Inclusion of visual information often does not improve performance, indicating a gap between visual content understanding and use for humor.
• LLMs can match human judgments in controlled settings but struggle with subtle differences in wit among semantically similar candidates.
• Preference-based selection generally outperforms similarity-based methods in raw scores, yet gains are limited overall.
• Performance degrades when candidates become more semantically similar, suggesting challenges in fine-grained humor discrimination.

Figure 2: Crowdworker annotation interface and full task instruction for the funniness scoring task in MaMe-Re . Top: interface screenshot. Bottom: instruction text shown to annotators.