[Paper Review] ADVISE: Symbolism and External Knowledge for Decoding Advertisements

Keren Ye, Adriana Kovashka|arXiv (Cornell University)|Nov 17, 2017

Multimodal Machine Learning Applications3 citations

TL;DR

ADVISE proposes a vision-and-language representation model that leverages symbolic associations (e.g., gun = danger, motorcycle = adventure) and external knowledge from object detection and image captioning to improve understanding of public service announcement (PSA) and commercial advertisements. By integrating symbol labels and object predictions as constraints and additive components in a joint image-text embedding space, the method achieves a 21% relative improvement over the state of the art in ad understanding, with strong zero-shot performance on slogan retrieval and clustering tasks.

ABSTRACT

In order to convey the most content in their limited space, advertisements embed references to outside knowledge via symbolism. For example, a motorcycle stands for adventure (a positive property the ad wants associated with the product being sold), and a gun stands for danger (a negative property to dissuade viewers from undesirable behaviors). We show how to use symbolic references to better understand the meaning of an ad. We further show how anchoring ad understanding in general-purpose object recognition and image captioning improves results. We formulate the ad understanding task as matching the ad image to human-generated statements that describe the action that the ad prompts, and the rationale it provides for taking this action. Our proposed method outperforms the state of the art on this task, and on an alternative formulation of question-answering on ads. We show additional applications of our learned representations for matching ads to slogans, and clustering ads according to their topic, without extra training.

Motivation & Objective

• To improve the understanding of advertisements—particularly conceptually complex public service announcements (PSAs)—by modeling symbolic references and external knowledge.
• To address the challenge of decoding non-literal, rhetorical messages in ads using visual-semantic embeddings that incorporate symbolic mappings and object-level recognition.
• To demonstrate that combining symbolic knowledge with modern object detection and attention mechanisms improves performance beyond standard visual-semantic embedding methods.
• To show that learned representations from ADVISE generalize to zero-shot tasks such as slogan retrieval and topic-based clustering without additional training.

Proposed method

• The method learns a joint image-text embedding space where ad images are mapped to statements describing the ad's intended action and rationale, using a triplet ranking loss.
• Image representations are built from region features extracted via region proposal networks (e.g., Faster R-CNN), with attention mechanisms assigning importance weights to each region.
• Symbolic associations (e.g., 'gun' → 'danger') are used as constraints to regularize the embedding space, ensuring that images with similar symbolic meanings are closer in the embedding space.
• Object detection predictions are mapped to symbolic concepts via a domain-specific knowledge base, which is used to refine the image representation through an additive component.
• The model incorporates noisy image captions as a source of external knowledge, propagating words from captions to image regions to improve semantic alignment.
• The final image representation combines region features, symbol-based constraints, and knowledge-enhanced embeddings, enabling robust zero-shot transfer to downstream tasks.

Experimental results

Research questions

• RQ1 Can symbolic associations (e.g., a gun symbolizing danger) be effectively leveraged to improve the understanding of non-literal, conceptual messages in public service advertisements?
• RQ2 How does integrating external knowledge from object detection and image captioning improve the performance of visual-semantic embeddings in ad understanding tasks?
• RQ3 To what extent do symbolic constraints and knowledge-augmented image representations generalize to zero-shot tasks such as slogan retrieval and topic clustering?
• RQ4 Does the use of modern object recognition techniques (e.g., region proposals and attention) significantly improve ad understanding compared to traditional recognition pipelines?

Key findings

• ADVISE outperforms the state of the art by 21% on the primary ad understanding task of matching ads to human-generated statements describing their message and rationale.
• On the question-answering classification task, ADVISE achieves 10.94% top-1 accuracy on PSAs and 12.64% on product ads—representing a 9% and 6% improvement, respectively, over the prior SOTA method.
• The method achieves a rank of 4.827 (±0.025) on the hard statement retrieval task, significantly outperforming VSE++ (5.635) and Hussain-Ranking (5.595), demonstrating superior zero-shot generalization.
• For slogan retrieval, ADVISE achieves a rank of 3.331 (±0.077), a substantial improvement over the next best method, indicating strong transferability of the learned representations.
• In topic-based clustering, ADVISE achieves a homogeneity score of 0.355 (±0.001), significantly higher than VSE++ (0.292) and Hussain-Ranking (0.291), showing better semantic clustering of ads by topic.
• The method demonstrates that conceptual knowledge (symbols) is more beneficial for PSAs, while general-purpose object recognition is more effective for product ads, highlighting task-specific utility of external knowledge sources.