[Paper Review] Video (GIF) Sentiment Analysis using Large-Scale Mid-Level Ontology

Zheng Cai, Donglin Cao+1 more|arXiv (Cornell University)|Jun 2, 2015

Sentiment Analysis and Opinion Mining6 references4 citations

TL;DR

This paper proposes SentiPair Sequence, a mid-level visual sentiment ontology built on a Synset Forest to model spatio-temporal abstractions in GIF videos for sentiment analysis. By combining adjective-noun (ANP) and verb-noun (VNP) pairs into ordered sequences, the method achieves 72.6% accuracy in sentiment prediction, outperforming ANP and VNP baselines. The authors also release the GSO-2015 dataset with 6,000 manually annotated GIFs for public use.

ABSTRACT

With faster connection speed, Internet users are now making social network a huge reservoir of texts, images and video clips (GIF). Sentiment analysis for such online platform can be used to predict political elections, evaluates economic indicators and so on. However, GIF sentiment analysis is quite challenging, not only because it hinges on spatio-temporal visual contentabstraction, but also for the relationship between such abstraction and final sentiment remains unknown.In this paper, we dedicated to find out such relationship.We proposed a SentiPairSequence basedspatiotemporal visual sentiment ontology, which forms the midlevel representations for GIFsentiment. The establishment process of SentiPair contains two steps. First, we construct the Synset Forest to define the semantic tree structure of visual sentiment label elements. Then, through theSynset Forest, we organically select and combine sentiment label elements to form a mid-level visual sentiment representation. Our experiments indicate that SentiPair outperforms other competing mid-level attributes. Using SentiPair, our analysis frameworkcan achieve satisfying prediction accuracy (72.6%). We also opened ourdataset (GSO-2015) to the research community. GSO-2015 contains more than 6,000 manually annotated GIFs out of more than 40,000 candidates. Each is labeled with both sentiment and SentiPair Sequence.

Motivation & Objective

• To address the challenge of GIF sentiment analysis, which is complicated by noise, irregular length, symbolism, and abstraction.
• To develop a mid-level visual representation that captures spatio-temporal sentiment abstractions in GIFs.
• To model the relationship between mid-level visual representations and final sentiment judgments.
• To create and release a large-scale, manually annotated GIF sentiment dataset (GSO-2015) to support future research.

Proposed method

• Construct a Synset Forest using WordNet to define a semantic tree structure for visual sentiment label elements.
• Generate SentiPairs by combining adjective-noun pairs (ANP) and verb-noun pairs (VNP) from the Synset Forest to form mid-level sentiment representations.
• Build the GIF Sentiment Ontology (GSO) by organizing SentiPairs into ordered sequences reflecting temporal occurrence in GIFs.
• Use crowd-sourced labeling (7 workers) to annotate 6,000+ GIFs from 40,000 candidates with both sentiment and SentiPair Sequences.
• Apply correlation-based feature selection and multiple classifiers (SMO, Logistic Regression, Random Forest) to evaluate performance.
• Train and test sentiment classifiers using the GSO-2015 dataset, comparing SentiPair Sequence against ANP and VNP baselines.

Experimental results

Research questions

• RQ1 How can we effectively represent high-level visual abstractions in GIF videos for sentiment analysis?
• RQ2 What is the relationship between mid-level visual representations and final sentiment judgments in GIFs?
• RQ3 Can a structured ontology of visual sentiment elements improve sentiment prediction accuracy compared to raw attributes?
• RQ4 How does incorporating temporal order of visual events (via SentiPair Sequence) enhance sentiment modeling?

Key findings

• SentiPair Sequence achieves a sentiment prediction accuracy of 72.6%, outperforming both ANP-only (70.2%) and VNP-only (65.8%) baselines.
• The use of SentiPair Sequences, which integrate motion (VNP) and static attributes (ANP), leads to better performance than using either type alone.
• Feature selection via correlation-based subset selection maintains high accuracy while reducing the number of mid-level attributes used.
• The GSO-2015 dataset, containing over 6,000 manually annotated GIFs with sentiment and SentiPair Sequences, is publicly released to support further research.
• The Synset Forest provides a scalable, semantically grounded framework for organizing visual sentiment concepts across multiple semantic dimensions.
• The framework demonstrates that modeling temporal sequences of visual sentiment elements significantly improves sentiment recognition in short video clips.