[Paper Review] Label Distribution Amendment with Emotional Semantic Correlations for Facial Expression Recognition

Shasha Mao, Guanghui Shi+5 more|arXiv (Cornell University)|Jul 23, 2021

Emotion and Mood Recognition31 references4 citations

TL;DR

This paper proposes a novel label distribution amendment method for facial expression recognition that leverages emotional semantic correlations to improve label reliability. By constructing a semantic class-relation graph and comparing it with a task-space class-relation graph, the method dynamically adjusts label distributions based on confidence, achieving superior performance on wild expression datasets like RAF-DB, especially under noisy or ambiguous labeling conditions.

ABSTRACT

By utilizing label distribution learning, a probability distribution is assigned for a facial image to express a compound emotion, which effectively improves the problem of label uncertainties and noises occurred in one-hot labels. In practice, it is observed that correlations among emotions are inherently different, such as surprised and happy emotions are more possibly synchronized than surprised and neutral. It indicates the correlation may be crucial for obtaining a reliable label distribution. Based on this, we propose a new method that amends the label distribution of each facial image by leveraging correlations among expressions in the semantic space. Inspired by inherently diverse correlations among word2vecs, the topological information among facial expressions is firstly explored in the semantic space, and each image is embedded into the semantic space. Specially, a class-relation graph is constructed to transfer the semantic correlation among expressions into the task space. By comparing semantic and task class-relation graphs of each image, the confidence of its label distribution is evaluated. Based on the confidence, the label distribution is amended by enhancing samples with higher confidence and weakening samples with lower confidence. Experimental results demonstrate the proposed method is more effective than compared state-of-the-art methods.

Motivation & Objective

• To address the limitations of one-hot labeling in facial expression recognition, particularly label uncertainty and noise in real-world (wild) datasets.
• To model intrinsic, diverse correlations among facial expressions—such as the co-occurrence likelihood of surprised and happy emotions—using semantic space representations.
• To improve label distribution reliability by dynamically amending distributions based on confidence derived from comparing semantic and task-space class-relation graphs.
• To demonstrate that incorporating emotional semantic correlations leads to more accurate and interpretable label distributions, especially for compound or ambiguous expressions.

Proposed method

• An auto-encoder is used to embed facial images into a semantic space, preserving topological relationships among facial expressions.
• A semantic class-relation graph is constructed based on learned semantic embeddings to encode intrinsic emotional correlations (e.g., surprised-happy vs. surprised-neutral).
• A task-space class-relation graph is learned from the model's predictions during training, reflecting the current confidence in category assignments.
• Confidence for each sample is evaluated by comparing the semantic and task-space class-relation graphs, identifying samples with consistent or inconsistent correlations.
• Label distributions are amended by boosting features and probabilities of high-confidence samples and suppressing low-confidence ones via a confidence-aware loss function.
• The method uses a hyperparameter β to balance the influence of the original label distribution and the amended distribution, with optimal values found empirically under varying noise levels.

Experimental results

Research questions

• RQ1 How can emotional semantic correlations between facial expressions be effectively modeled to improve label distribution quality in facial expression recognition?
• RQ2 To what extent can semantic correlations reduce the impact of label noise and uncertainty in wild expression datasets?
• RQ3 Can a confidence score derived from comparing semantic and task-space class-relation graphs reliably identify ambiguous or mislabeled facial expressions?
• RQ4 How does the proposed amendment of label distributions affect the performance of deep learning models in facial expression recognition?
• RQ5 Can the method generate more interpretable and realistic label distributions for compound expressions than baseline one-hot or standard label distribution learning approaches?

Key findings

• The proposed method, SCE-ALD, achieves state-of-the-art performance on three wild expression databases (RAF-DB, AffectNet, FER2013), outperforming existing label distribution learning methods.
• On RAF-DB with 30% synthetic label noise, the method achieves the highest accuracy when β = 0.2, indicating that low β values are optimal under high noise conditions.
• The method demonstrates robustness to label noise, with performance improvements most significant in high-noise regimes, confirming the importance of confidence-based label amendment.
• Visualization results show that the amended label distributions are semantically plausible: for example, happy is assigned higher probability than disgusted in a compound expression where happiness is visually dominant.
• In compound RAF-DB, the model’s predicted label distributions align closely with human-annotated multi-labels, and the probabilities reflect visual salience—e.g., assigning higher probability to sadness than fear when sadness is more prominent.
• Ablation studies confirm that the label distribution amendment mechanism is indispensable, as the baseline method (β = 1) performs worse than the proposed adaptive amendment strategy.