[Paper Review] Modeling Emotion Influence from Images in Social Networks

Xiaohui Wang, Jia Jia+2 more|arXiv (Cornell University)|Jan 17, 2014

Image Retrieval and Classification Techniques14 references3 citations

TL;DR

This paper proposes a probabilistic factor graph model that captures emotion influence from images in social networks by integrating visual content, temporal emotion patterns, and social network effects. The model improves emotion prediction accuracy by 5% over SVM, demonstrating that social influence significantly enhances image-based emotion classification in platforms like Flickr.

ABSTRACT

Images become an important and prevalent way to express users' activities, opinions and emotions. In a social network, individual emotions may be influenced by others, in particular by close friends. We focus on understanding how users embed emotions into the images they uploaded to the social websites and how social influence plays a role in changing users' emotions. We first verify the existence of emotion influence in the image networks, and then propose a probabilistic factor graph based emotion influence model to answer the questions of "who influences whom". Employing a real network from Flickr as experimental data, we study the effectiveness of factors in the proposed model with in-depth data analysis. Our experiments also show that our model, by incorporating the emotion influence, can significantly improve the accuracy (+5%) for predicting emotions from images. Finally, a case study is used as the anecdotal evidence to further demonstrate the effectiveness of the proposed model.

Motivation & Objective

• To investigate whether emotion influence exists in image-based social networks like Flickr.
• To develop a principled probabilistic model that captures how users' emotions are influenced by friends' uploaded images.
• To improve the accuracy of emotion prediction from images by incorporating social influence, visual features, and temporal dynamics.
• To validate the model's effectiveness through empirical evaluation on real Flickr data and a case study.

Proposed method

• A probabilistic factor graph model is proposed to jointly model emotion influence, integrating three key factors: content correlation (visual features), temporal correlation (past emotions), and social correlation (friend influence).
• The model uses factor functions to represent the influence of friends' emotions on a user’s emotion, with specific functions for immediate, stable, and transient influence patterns.
• Emotion prediction is performed using maximum a posteriori (MAP) estimation over the factor graph, combining evidence from images, user history, and social network structure.
• The model is trained and evaluated on a real Flickr dataset containing 1.25 million images from 4,725 users over 12 weeks.
• Feature selection and ablation studies are conducted to assess the contribution of each factor (visual, temporal, social) to prediction performance.

Experimental results

Research questions

• RQ1 Does emotion influence exist in image-based social networks, where users share images to express emotions?
• RQ2 How can social influence among users be modeled and quantified when emotions are expressed through images?
• RQ3 To what extent do visual features, temporal emotion patterns, and social network structure improve emotion prediction accuracy?
• RQ4 Which social influence patterns (e.g., immediate, stable, transient) contribute most to emotion prediction?

Key findings

• A matched sampling test confirms that when a user has more than two friends who uploaded 'happy' images, the likelihood of that user also uploading a 'happy' image nearly doubles.
• The proposed model improves emotion prediction accuracy by 5% compared to SVM, with a significant performance gain in F1-measure across all emotion categories.
• The 'stable influence' factor (f5) contributes the most to performance, indicating that emotional influence persists over time and is not short-lived.
• Removing the social influence factor (Model-f4) results in the largest performance drop, confirming that social correlation is the most impactful factor in the model.
• The case study of user Mike shows that the model correctly infers bidirectional emotion influence (e.g., from Mike to friend d) that aligns with visual and temporal observations of image content and timing.