[Paper Review] Deep Covariance Descriptors for Facial Expression Recognition

Naima Otberdout, Anis Kacem+3 more|arXiv (Cornell University)|May 10, 2018

Emotion and Mood RecognitionPsychology27 references19 citations

TL;DR

This paper proposes using deep covariance descriptors—encoding features from VGG-face and ExpNet as symmetric positive definite (SPD) matrices—on the Riemannian manifold for facial expression recognition. By applying a Gaussian RBF kernel on the SPD manifold, the method achieves state-of-the-art performance on Oulu-CASIA and SFEW datasets, outperforming standard softmax classification with fully connected layers.

ABSTRACT

In this paper, covariance matrices are exploited to encode the deep convolutional neural networks (DCNN) features for facial expression recognition. The space geometry of the covariance matrices is that of Symmetric Positive Definite (SPD) matrices. By performing the classification of the facial expressions using Gaussian kernel on SPD manifold, we show that the covariance descriptors computed on DCNN features are more efficient than the standard classification with fully connected layers and softmax. By implementing our approach using the VGG-face and ExpNet architectures with extensive experiments on the Oulu-CASIA and SFEW datasets, we show that the proposed approach achieves performance at the state of the art for facial expression recognition.

Motivation & Objective

• To improve facial expression recognition by encoding deep CNN features into compact, discriminative representations that capture spatial relationships.
• To overcome the limitation of standard DCNNs in modeling spatial dependencies within facial regions.
• To explore the geometric structure of covariance matrices as points on the symmetric positive definite (SPD) manifold for improved classification.
• To evaluate the proposed method on benchmark datasets and compare it with state-of-the-art approaches.
• To investigate the effectiveness of both global and local covariance descriptors in capturing expression-specific patterns.

Proposed method

• Extract deep convolutional features from pre-trained VGG-face and ExpNet models on facial images.
• Compute global covariance descriptors by aggregating 512 feature maps into a single SPD matrix per image.
• Detect local facial regions using landmarks and extract region-specific feature maps for local covariance descriptors.
• Represent each local region’s features as a local covariance matrix, forming a set of SPD matrices.
• Apply a valid positive definite Gaussian RBF kernel on the SPD manifold to enable SVM-based classification.
• Train and evaluate the classifier using the Riemannian geometry of SPD matrices, leveraging the manifold's intrinsic structure.

Experimental results

Research questions

• RQ1 Can covariance descriptors of deep CNN features improve facial expression recognition compared to standard softmax classification?
• RQ2 How does the geometric structure of the SPD manifold enhance classification performance on facial expression data?
• RQ3 Does combining global and local covariance descriptors yield better performance than using either alone?
• RQ4 How does the proposed method compare to state-of-the-art approaches on standard benchmarks like Oulu-CASIA and SFEW?
• RQ5 What is the impact of landmark detection reliability on local covariance descriptor quality, especially on challenging datasets like SFEW?

Key findings

• The proposed method achieved state-of-the-art performance on the Oulu-CASIA dataset, outperforming previous methods using the same evaluation protocol.
• On the SFEW dataset, the method achieved the second-highest accuracy, surpassing multiple state-of-the-art approaches with significant gains.
• The use of Gaussian RBF kernel on the SPD manifold led to better classification performance than standard softmax with fully connected layers.
• Visualizations showed consistent color patterns in covariance descriptors for similar expressions (e.g., pink/purple for happy, blue for surprise), indicating discriminative feature learning.
• Local covariance descriptors captured distinct patterns per facial region, suggesting effective spatial modeling.
• Failure in facial landmark detection on ~30% of SFEW images limited the benefit of combining local and global descriptors on that dataset.