[Paper Review] A Circular-Structured Representation for Visual Emotion Distribution Learning

Jingyuan Yang, Jie Li+3 more|arXiv (Cornell University)|Jun 23, 2021

Sentiment Analysis and Opinion Mining46 references4 citations

TL;DR

This paper proposes a circular-structured representation for visual emotion distribution learning, modeling emotions as vectors on an Emotion Circle defined by polarity, type, and intensity, and introduces a Progressive Circular (PC) loss to refine predictions in a coarse-to-fine manner. The method outperforms state-of-the-art approaches on three public datasets by effectively leveraging psychological priors on emotional relationships.

ABSTRACT

Visual Emotion Analysis (VEA) has attracted increasing attention recently with the prevalence of sharing images on social networks. Since human emotions are ambiguous and subjective, it is more reasonable to address VEA in a label distribution learning (LDL) paradigm rather than a single-label classification task. Different from other LDL tasks, there exist intrinsic relationships between emotions and unique characteristics within them, as demonstrated in psychological theories. Inspired by this, we propose a well-grounded circular-structured representation to utilize the prior knowledge for visual emotion distribution learning. To be specific, we first construct an Emotion Circle to unify any emotional state within it. On the proposed Emotion Circle, each emotion distribution is represented with an emotion vector, which is defined with three attributes (i.e., emotion polarity, emotion type, emotion intensity) as well as two properties (i.e., similarity, additivity). Besides, we design a novel Progressive Circular (PC) loss to penalize the dissimilarities between predicted emotion vector and labeled one in a coarse-to-fine manner, which further boosts the learning process in an emotion-specific way. Extensive experiments and comparisons are conducted on public visual emotion distribution datasets, and the results demonstrate that the proposed method outperforms the state-of-the-art methods.

Motivation & Objective

• To address the limitations of single-label classification in Visual Emotion Analysis (VEA), which oversimplifies the ambiguity and subjectivity of human emotions.
• To model intrinsic emotional relationships—such as polarity, type, and intensity—using psychological theories like Mikel’s Wheel.
• To develop a novel representation that unifies basic and compound emotions within a circular structure for more accurate distribution learning.
• To design a Progressive Circular (PC) loss that enforces emotion-specific constraints in a coarse-to-fine manner, improving prediction fidelity.
• To demonstrate superior performance over state-of-the-art methods on standard visual emotion distribution datasets.

Proposed method

• Constructs an Emotion Circle where eight basic emotions are represented as unit vectors evenly distributed, forming a circular structure based on psychological models.
• Represents any emotion distribution as a compound emotion vector via weighted combination of basic emotion vectors, with attributes: polarity, type, and intensity.
• Defines two key properties: similarity (angular proximity on the circle) and additivity (vector sum consistency) to preserve emotional relationships.
• Proposes a Progressive Circular (PC) loss that applies three constraints—polar loss, type loss, and intensity loss—sequentially to guide prediction refinement.
• Combines the PC loss with standard Kullback-Leibler (KL) divergence loss, with a hyperparameter μ=0.7 to balance emotion-specific and general distribution learning.
• Uses a multi-head network head to predict emotion vectors and applies the PC loss during training to enforce structural and semantic consistency.

Experimental results

Research questions

• RQ1 How can intrinsic emotional relationships—such as polarity, type, and intensity—be effectively modeled in visual emotion distribution learning?
• RQ2 Can a circular-structured representation unify both basic and compound emotional states in a psychologically grounded way?
• RQ3 Does a progressive, coarse-to-fine loss function that enforces polarity, type, and intensity constraints improve prediction accuracy compared to standard losses?
• RQ4 How does the integration of psychological priors (e.g., Mikel’s Wheel) enhance performance in visual emotion distribution learning?
• RQ5 What is the optimal balance between the proposed PC loss and standard KL divergence loss for emotion distribution learning?

Key findings

• The proposed method achieves consistent performance gains over state-of-the-art methods across three public datasets: Flickr_LDL, Twitter_LDL, and AffectNet-Emotion.
• The Progressive Circular (PC) loss significantly improves performance when all three constraints—polarity, type, and intensity—are applied, with each contributing independently and progressively.
• The ablation study confirms that each component of the PC loss is indispensable, as removing any constraint leads to performance degradation.
• The optimal hyperparameter μ=0.7 balances the PC loss and KL loss, as performance drops when μ is too high or too low.
• Visualization results show that the model accurately recovers both simple and complex emotion distributions, especially in preserving polarity and intensity.
• Top-1 accuracy improves with the inclusion of PC loss, indicating better prediction of dominant emotions in distributions.