[Paper Review] A Multimodal Analysis of Influencer Content on Twitter

Danae Sánchez Villegas, Cătălina Goanță+1 more|arXiv (Cornell University)|Sep 6, 2023

Sentiment Analysis and Opinion Mining4 citations

TL;DR

This paper introduces a new, expert-annotated Twitter dataset of 15,998 influencer posts labeled as commercial or non-commercial, and proposes a cross-attention multimodal model (ViT-BERTweet-Att) that outperforms state-of-the-art models in detecting hidden commercial content by combining text and image features, significantly reducing false positives and improving detection of undisclosed sponsorships.

ABSTRACT

Influencer marketing involves a wide range of strategies in which brands collaborate with popular content creators (i.e., influencers) to leverage their reach, trust, and impact on their audience to promote and endorse products or services. Because followers of influencers are more likely to buy a product after receiving an authentic product endorsement rather than an explicit direct product promotion, the line between personal opinions and commercial content promotion is frequently blurred. This makes automatic detection of regulatory compliance breaches related to influencer advertising (e.g., misleading advertising or hidden sponsorships) particularly difficult. In this work, we (1) introduce a new Twitter (now X) dataset consisting of 15,998 influencer posts mapped into commercial and non-commercial categories for assisting in the automatic detection of commercial influencer content; (2) experiment with an extensive set of predictive models that combine text and visual information showing that our proposed cross-attention approach outperforms state-of-the-art multimodal models; and (3) conduct a thorough analysis of strengths and limitations of our models. We show that multimodal modeling is useful for identifying commercial posts, reducing the amount of false positives, and capturing relevant context that aids in the discovery of undisclosed commercial posts.

Motivation & Objective

• To address the challenge of detecting undisclosed commercial content in influencer posts, where promotional messages are often disguised as personal opinions.
• To create a large-scale, expert-annotated dataset of Twitter posts labeled for commercial vs. non-commercial intent to support automatic detection research.
• To evaluate the effectiveness of multimodal approaches—combining text and visual features—in distinguishing between authentic personal content and hidden product promotions.
• To investigate the limitations of existing models and identify when text, vision, or multimodal fusion is most effective for commercial content detection.
• To provide a foundation for future work in multilingual and cross-cultural analysis of influencer advertising compliance.

Proposed method

• Curated a new dataset of 15,998 influencer tweets from Twitter (now X), including 14,384 text-image pairs and 1,614 text-only posts, expert-annotated into commercial and non-commercial categories.
• Employed a cross-attention mechanism to fuse representations from a vision transformer (ViT) and a fine-tuned BERTweet model, enabling dynamic interaction between modalities.
• Benchmarked the proposed model against state-of-the-art vision, language, and multimodal models, including LLM prompting with zero-shot and few-shot prompting strategies.
• Used keyword matching and domain-specific term lists (e.g., #ad, #sponsored) as a baseline, but focused on learning contextual and stylistic cues beyond explicit disclosure terms.
• Conducted qualitative analysis on misclassified examples to identify recurring error patterns and model limitations.
• Validated results on a time-extended dataset from January 2015 to August 2021, covering domains such as beauty, travel, food, fitness, technology, and lifestyle.

Figure 3: Example of Annotation

Experimental results

Research questions

• RQ1 How effective is multimodal modeling in distinguishing commercial from non-commercial influencer content compared to unimodal or keyword-based approaches?
• RQ2 What are the main failure modes of current models in detecting hidden commercial posts, particularly when disclosures are missing or ambiguous?
• RQ3 In what contexts do visual features contribute more than textual cues, and vice versa, for commercial content detection?
• RQ4 Can a cross-attention mechanism improve performance over standard late or early fusion strategies in multimodal influencer content classification?
• RQ5 How do stylistic and contextual cues in text and image combinations help detect undisclosed sponsorships beyond keyword matching?

Key findings

• The proposed cross-attention multimodal model, ViT-BERTweet-Att, outperforms all baseline models, including state-of-the-art multimodal architectures, in detecting commercial influencer content.
• Multimodal modeling reduces false positives by 85% compared to keyword-based baselines, which often misclassify non-commercial posts containing promotional keywords.
• The model successfully identifies undisclosed commercial posts in 15% of cases where text-only models fail, particularly when brand mentions are absent but visual cues suggest product promotion.
• The most common error type (20%) involves posts describing personal experiences—especially travel-related content—without brand names or explicit disclosures, which are challenging for both text and multimodal models.
• Text-only models frequently misclassify non-commercial content as commercial due to the presence of promotional keywords used in non-promotional contexts, highlighting the need for contextual understanding.
• The model demonstrates that visual features and stylistic cues in text (e.g., product placement, aesthetic framing) are critical for detecting native advertising disguised as personal content.

Figure 4: ViT-BERTweet-Att model for detecting commercial content. FC: fully-connected layer.