[Paper Review] Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text

Mingda Zhang, Rebecca Hwa+1 more|arXiv (Cornell University)|Sep 1, 2018

Language, Metaphor, and Cognition10 citations

TL;DR

This paper investigates non-literal, implicit relationships between persuasive images and text in advertisements, proposing a method to classify whether image-text pairs are parallel (conveying the same message non-literally) or non-parallel. Using a crowdsourced dataset and features capturing image creativity and text specificity/ambiguity, the approach outperforms standard image-text alignment methods in predicting parallelism.

ABSTRACT

Images and text in advertisements interact in complex, non-literal ways. The two channels are usually complementary, with each channel telling a different part of the story. Current approaches, such as image captioning methods, only examine literal, redundant relationships, where image and text show exactly the same content. To understand more complex relationships, we first collect a dataset of advertisement interpretations for whether the image and slogan in the same visual advertisement form a parallel (conveying the same message without literally saying the same thing) or non-parallel relationship, with the help of workers recruited on Amazon Mechanical Turk. We develop a variety of features that capture the creativity of images and the specificity or ambiguity of text, as well as methods that analyze the semantics within and across channels. We show that our method outperforms standard image-text alignment approaches on predicting the parallel/non-parallel relationship between image and text.

Motivation & Objective

• To understand complex, non-literal relationships between persuasive images and text in advertisements beyond literal alignment.
• To address the limitation of existing image captioning methods, which only model redundant, literal image-text pairs.
• To develop a classification framework that distinguishes between parallel (non-literal similarity) and non-parallel image-text relationships in advertising.
• To collect and analyze a dataset of human-annotated interpretations of image-slogan relationships to capture implicit semantic connections.

Proposed method

• Collect a dataset of advertisement interpretations via Amazon Mechanical Turk, labeling image-slogan pairs as parallel or non-parallel.
• Design features that capture the creativity of images and the specificity or ambiguity of text to model non-literal relationships.
• Apply semantic analysis techniques to examine meaning within and across image and text modalities.
• Train a classification model using these features to predict whether an image-slogan pair forms a parallel relationship.
• Use cross-modal semantic embeddings to compare and align image and text representations beyond literal content matching.

Experimental results

Research questions

• RQ1 How do people perceive the relationship between persuasive images and slogans in advertisements beyond literal content matching?
• RQ2 What features best capture the implicit, non-literal connection between image and text in advertising?
• RQ3 Can a machine learning model effectively classify image-slogan pairs as parallel or non-parallel based on semantic and stylistic features?
• RQ4 How does the proposed method compare to standard image-text alignment approaches in predicting parallelism?

Key findings

• The proposed method achieves superior performance in predicting parallel vs. non-parallel image-text relationships compared to standard image-text alignment techniques.
• Features capturing image creativity and text specificity/ambiguity significantly improve classification performance.
• Human-annotated interpretations reveal that parallel relationships in ads often rely on metaphorical or symbolic meaning rather than literal content.
• The dataset of annotated image-slogan pairs provides a valuable resource for studying implicit image-text relationships in advertising.