[Paper Review] Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks

Quanzeng You, Jiebo Luo+2 more|arXiv (Cornell University)|Sep 20, 2015

Sentiment Analysis and Opinion MiningComputer Science27 references213 citations

TL;DR

This paper proposes a deep convolutional neural network (CNN) framework for robust image sentiment analysis using progressively trained and domain-transferred models. By leveraging half a million weakly labeled Flickr images and fine-tuning on a small set of manually labeled Twitter images, the method achieves state-of-the-art performance, with a 0.846 F1-score on Twitter test data, significantly outperforming baseline methods using handcrafted features or pre-trained models.

ABSTRACT

Sentiment analysis of online user generated content is important for many social media analytics tasks. Researchers have largely relied on textual sentiment analysis to develop systems to predict political elections, measure economic indicators, and so on. Recently, social media users are increasingly using images and videos to express their opinions and share their experiences. Sentiment analysis of such large scale visual content can help better extract user sentiments toward events or topics, such as those in image tweets, so that prediction of sentiment from visual content is complementary to textual sentiment analysis. Motivated by the needs in leveraging large scale yet noisy training data to solve the extremely challenging problem of image sentiment analysis, we employ Convolutional Neural Networks (CNN). We first design a suitable CNN architecture for image sentiment analysis. We obtain half a million training samples by using a baseline sentiment algorithm to label Flickr images. To make use of such noisy machine labeled data, we employ a progressive strategy to fine-tune the deep network. Furthermore, we improve the performance on Twitter images by inducing domain transfer with a small number of manually labeled Twitter images. We have conducted extensive experiments on manually labeled Twitter images. The results show that the proposed CNN can achieve better performance in image sentiment analysis than competing algorithms.

Motivation & Objective

• To address the challenge of limited manually labeled visual sentiment data by leveraging large-scale, noisy image datasets.
• To improve model generalization across domains, especially from Flickr to Twitter, using transfer learning.
• To develop a deep learning framework that outperforms traditional methods relying on handcrafted visual features or attributes.
• To enable effective sentiment prediction from visual content, complementing textual sentiment analysis in social media analytics.
• To demonstrate the feasibility and effectiveness of using deep CNNs with progressive training and domain adaptation for visual sentiment analysis.

Proposed method

• A custom deep CNN architecture is designed specifically for image sentiment analysis, optimized for high-level abstraction and sentiment classification.
• A progressive training strategy is employed to fine-tune the CNN on a large set of half a million weakly labeled Flickr images, improving robustness to noisy labels.
• Domain transfer is applied using only 1269 manually labeled Twitter images to adapt the pre-trained model to the target domain.
• The model is fine-tuned end-to-end on the target domain data, allowing knowledge transfer and improved performance on unseen Twitter images.
• Performance is evaluated using standard metrics including precision, recall, and F1-score across multiple test sets.
• The framework enables transfer learning with minimal labeled data in the target domain, reducing reliance on large-scale manual annotation.

Experimental results

Research questions

• RQ1 Can a deep CNN trained on large-scale, noisy image data achieve robust performance in image sentiment analysis?
• RQ2 How effective is progressive training in mitigating the impact of noisy labels in weakly supervised learning?
• RQ3 To what extent can a model trained on Flickr images generalize to Twitter images using only a small number of manually labeled examples?
• RQ4 Does end-to-end fine-tuning with domain-specific data improve sentiment classification accuracy compared to fixed feature-based models?
• RQ5 Can deep neural networks outperform traditional methods using handcrafted low-level or mid-level visual features for visual sentiment analysis?

Key findings

• The proposed CNN achieved an F1-score of 0.846 on the Twitter test set, significantly outperforming all baseline methods, including SentiBank (0.776) and Sentribute (0.805).
• The fine-tuned CNN model achieved a precision of 0.905 and recall of 0.855 on the Twitter test set, indicating strong performance on positive sentiment prediction.
• Progressive training on noisy Flickr data followed by domain adaptation on 1269 Twitter images led to a notable improvement over models trained only on source-domain data.
• Both the fine-tuned CNN and PCNN models achieved similar performance, suggesting that the fine-tuning process helps converge to effective local minima despite different initialization.
• Top-ranked predictions from both models were correctly classified, but the images predicted as top positives were not identical, indicating different learned representations due to distinct optimization paths.
• Common misclassifications were observed among models using low-level features, suggesting shared limitations in feature representation, while deep learning models showed more consistent and robust performance.