[Paper Review] Efficient Facial Expression Analysis For Dimensional Affect Recognition Using Geometric Features

Vassilios Vonikakis, Stefan Winkler|arXiv (Cornell University)|Jun 15, 2021

Emotion and Mood Recognition22 references4 citations

TL;DR

This paper proposes a computationally efficient facial expression analysis system for dimensional affect recognition using geometric features and Partial Least Squares (PLS) regression. It jointly estimates Arousal, Valence, and Intensity from 2D facial landmarks, achieving performance comparable to deep learning models with significantly reduced computational cost and offering intuitive interpretability of learned features.

ABSTRACT

Despite their continued popularity, categorical approaches to affect recognition have limitations, especially in real-life situations. Dimensional models of affect offer important advantages for the recognition of subtle expressions and more fine-grained analysis. We introduce a simple but effective facial expression analysis (FEA) system for dimensional affect, solely based on geometric features and Partial Least Squares (PLS) regression. The system jointly learns to estimate Arousal and Valence ratings from a set of facial images. The proposed approach is robust, efficient, and exhibits comparable performance to contemporary deep learning models, while requiring a fraction of the computational resources.

Motivation & Objective

• Address the limitations of categorical facial expression recognition in real-world, low-intensity scenarios by adopting dimensional affect models.
• Develop a lightweight, efficient FEA system that operates effectively under unconstrained (‘in the wild’) conditions.
• Enable joint estimation of Arousal, Valence, and Intensity using only geometric facial features.
• Ensure model interpretability by leveraging distances between facial landmarks as input features.
• Achieve performance on par with deep learning models while requiring a fraction of the computational resources.

Proposed method

• Use the Supervised Descent Method (SDM) to detect 2D facial landmarks from input images.
• Apply landmark-based face frontalization to normalize head pose variations, improving robustness to non-frontal views.
• Extract 1176 pairwise distances between 49 facial landmarks as geometric features for affect representation.
• Train a Partial Least Squares (PLS) regression model to jointly predict Arousal, Valence, and Intensity from geometric features.
• Leverage the MorphSet framework for data augmentation, transforming categorical datasets into balanced, dimensional AVI-annotated datasets.
• Select PLS with 29 components as the final model due to its superior generalization and stability across diverse real-world data distributions.

Experimental results

Research questions

• RQ1 Can a geometric feature-based approach achieve competitive performance in dimensional affect recognition compared to deep learning models?
• RQ2 How does the proposed system perform under real-world conditions with varying lighting and pose?
• RQ3 To what extent do the learned geometric features align with psychophysically observed facial deformation patterns?
• RQ4 Does PLS regression offer better generalization than alternative regression methods like Ridge or PCR in out-of-distribution settings?
• RQ5 Can the model’s predictions be meaningfully interpreted in terms of facial anatomy and affective meaning?

Key findings

• The PLS-based system achieved a mean squared error (MSE) of 0.0314 for Arousal, 0.0388 for Valence, and 0.0281 for Intensity on the test set, demonstrating performance comparable to deep learning models.
• Among all regression methods tested, PLS with 29 components showed the most stable and robust performance during qualitative testing on live webcam feeds, indicating superior generalization to unseen data distributions.
• The top contributing geometric features for Arousal were distances involving the inner parts of the eyebrows relative to the eyes, consistent with psychophysical observations of high-arousal expressions.
• Increased eccentricity of the mouth corners and upward movement of the inner eyebrows were strongly associated with higher Valence, aligning with findings on genuine vs. fake smiles.
• Intensity was predicted as a combination of features from both Arousal and Valence, particularly involving inner eyebrow-to-eye distances and mouth eccentricity, reflecting the magnitude of facial deformation.
• The PLS model’s weights enabled intuitive interpretation of facial features linked to affect, with thicker and darker lines in the visualization indicating higher contribution to the predicted affective values.