[Paper Review] SMP Challenge: An Overview of Social Media Prediction Challenge 2019

Bo Wu, Wen-Huang Cheng+4 more|arXiv (Cornell University)|Oct 4, 2019

Sentiment Analysis and Opinion Mining25 references4 citations

TL;DR

This paper introduces the SMP Challenge 2019, a large-scale social multimedia prediction challenge focused on Temporal Popularity Prediction (TPP), where teams predict future post popularity (e.g., likes, views) before publication. Using the newly released SMPD dataset of 486K posts from 80K users with rich visual, textual, and temporal metadata, the challenge evaluates models using Spearman correlation and mean absolute error, advancing predictive learning in social media analytics.

ABSTRACT

"SMP Challenge" aims to discover novel prediction tasks for numerous data on social multimedia and seek excellent research teams. Making predictions via social multimedia data (e.g. photos, videos or news) is not only helps us to make better strategic decisions for the future, but also explores advanced predictive learning and analytic methods on various problems and scenarios, such as multimedia recommendation, advertising system, fashion analysis etc. In the SMP Challenge at ACM Multimedia 2019, we introduce a novel prediction task Temporal Popularity Prediction, which focuses on predicting future interaction or attractiveness (in terms of clicks, views or likes etc.) of new online posts in social media feeds before uploading. We also collected and released a large-scale SMPD benchmark with over 480K posts from 69K users. In this paper, we define the challenge problem, give an overview of the dataset, present statistics of rich information for data and annotation and design the accuracy and correlation evaluation metrics for temporal popularity prediction to the challenge.

Motivation & Objective

• To establish a novel, large-scale benchmark for social multimedia prediction by introducing the Temporal Popularity Prediction task.
• To collect and release a diverse, multi-modal dataset (SMPD) with over 486,000 posts and 80,000 users to support real-world prediction research.
• To define standardized evaluation metrics—Spearman correlation and Mean Absolute Error—for assessing temporal popularity prediction performance.
• To foster innovation in predictive learning by challenging teams to forecast post popularity before publication using multi-modal content and temporal dynamics.
• To enable cross-task generalization by ensuring data diversity across users, geographic regions, and content modalities.

Proposed method

• The SMPD dataset is constructed with temporal ordering to preserve sequence continuity of social media posts.
• Each post includes visual content (images with semantic categories), textual content (titles and user-generated tags), and metadata (location, time, user profile).
• A log-normalization strategy is applied to balance post distribution across geographic territories, ensuring global representativeness.
• The evaluation uses a time-series partitioning strategy: 10-time-window sequences are split 2:1 for training and testing, preserving user-post sequences.
• Performance is evaluated using two metrics: Spearman Ranking Correlation (SRC) to measure ordinal ranking accuracy and Mean Absolute Error (MAE) to quantify prediction error.
• Final team rankings are computed by combining performance across both metrics to ensure balanced evaluation.

Experimental results

Research questions

• RQ1 What is the most effective way to predict the future popularity of social media posts before they are published, based on their content and context?
• RQ2 How do visual, textual, and temporal features collectively influence the predictability of post popularity in social media?
• RQ3 To what extent can multi-modal features (images, text, time) improve temporal popularity prediction compared to unimodal approaches?
• RQ4 How does geographic and user diversity in the dataset affect model generalization and performance across different social media contexts?
• RQ5 What evaluation metrics best reflect the quality of temporal popularity prediction in real-world social media systems?

Key findings

• The SMPD dataset contains 486,000 posts from 80,000 users, with rich multi-modal information including visual content, titles, tags, and temporal metadata.
• The dataset covers diverse geographic regions, with post distribution normalized via log-normalization to reduce regional bias.
• The popularity score distribution is highly skewed, with most posts receiving low engagement and a small fraction achieving high virality.
• The evaluation protocol uses a time-series partitioning strategy that maintains user-post sequence integrity, ensuring realistic temporal evaluation.
• Spearman correlation and Mean Absolute Error (MAE) are used as primary metrics, with final rankings based on combined performance across both.
• The challenge successfully enabled benchmarking of predictive models across diverse social media scenarios, advancing research in multimedia prediction and analytics.