[Paper Review] memeBot: Towards Automatic Image Meme Generation

Aadhavan Sadasivam, Kausic Gunasekar+2 more|arXiv (Cornell University)|Apr 30, 2020

Sentiment Analysis and Opinion Mining24 references4 citations

TL;DR

memeBot proposes an end-to-end neural encoder-decoder framework for automatic image meme generation, treating the task as a translation from text to a meme (image + caption). It selects a meme template and generates a caption conditioned on both the input sentence and template, achieving 66.5% confidence in coherence and 66.25% in relevance via human evaluation on Twitter data.

ABSTRACT

Image memes have become a widespread tool used by people for interacting and exchanging ideas over social media, blogs, and open messengers. This work proposes to treat automatic image meme generation as a translation process, and further present an end to end neural and probabilistic approach to generate an image-based meme for any given sentence using an encoder-decoder architecture. For a given input sentence, an image meme is generated by combining a meme template image and a text caption where the meme template image is selected from a set of popular candidates using a selection module, and the meme caption is generated by an encoder-decoder model. An encoder is used to map the selected meme template and the input sentence into a meme embedding and a decoder is used to decode the meme caption from the meme embedding. The generated natural language meme caption is conditioned on the input sentence and the selected meme template. The model learns the dependencies between the meme captions and the meme template images and generates new memes using the learned dependencies. The quality of the generated captions and the generated memes is evaluated through both automated and human evaluation. An experiment is designed to score how well the generated memes can represent the tweets from Twitter conversations. Experiments on Twitter data show the efficacy of the model in generating memes for sentences in online social interaction.

Motivation & Objective

• To address the challenge of generating image memes from arbitrary input sentences in online social interactions.
• To model meme generation as a translation task from natural language to a multimodal output (image + caption).
• To develop an end-to-end system that jointly selects a meme template and generates a contextually relevant caption.
• To evaluate the quality of generated memes using both automated metrics and human evaluation on real-world Twitter data.
• To enable controlled meme generation by conditioning captions on specific meme templates, even when not seen during training.

Proposed method

• Uses an encoder-decoder architecture with a joint embedding space to map input sentences and selected meme templates into a shared meme embedding.
• Employs a template selection module to choose the most suitable meme image from a curated set of popular templates based on input sentence semantics.
• Applies a transformer-based decoder to generate the meme caption conditioned on the input sentence and selected template embedding.
• Trains the model to maximize likelihood of correct template selection and accurate caption generation using a large-scale meme caption dataset.
• Incorporates data corruption during training to improve robustness and generalization to unseen input sentences.
• Enables controlled generation by conditioning the caption generator on a specified meme template, even if not in the training distribution.

Experimental results

Research questions

• RQ1 Can automatic image meme generation be effectively modeled as a neural machine translation task from text to multimodal output?
• RQ2 How well can a joint encoder-decoder model generate coherent and relevant meme captions conditioned on both input sentences and meme templates?
• RQ3 To what extent does the model generalize to unseen input sentences and maintain quality in human evaluation?
• RQ4 Can the model be controlled to generate captions for a specific meme template, even when not explicitly seen during training?
• RQ5 How do human raters perceive the quality and relevance of generated memes compared to human-created ones?

Key findings

• The model achieves 66.5% confidence in generating coherent meme captions and 66.25% confidence in generating relevant captions, based on human evaluation.
• 65% of raters liked the generated memes, with like scores strongly correlating with coherence and relevance ratings.
• Over 60% of generated memes were rated as coherent and relevant by human evaluators, indicating strong performance on the core quality dimensions.
• Inter-rater reliability was high, with Cohen’s Kappa scores above 60% for all metrics (coherence: 71.68%, relevance: 61.39%, user likes: 73.85%).
• The model successfully generalizes to generate captions conditioned on any given meme template, even when not in the training data, demonstrating robustness and control.
• The system effectively learns dependencies between meme captions, templates, and input sentences, enabling end-to-end generation of semantically aligned memes.