[Paper Review] Deep and Shallow Covariance Feature Quantization for 3D Facial Expression Recognition

Walid Hariri, Nadir Farah+1 more|arXiv (Cornell University)|May 12, 2021

Face and Expression Recognition54 references4 citations

TL;DR

This paper proposes a novel 3D facial expression recognition method using covariance feature quantization from both deep and shallow features extracted from 3D face scans and their 2D projections. By leveraging CNNs for deep features, geometric descriptors for shallow features, and applying covariance matrix learning followed by Bag-of-Features quantization, the approach achieves state-of-the-art accuracy on BU-3DFE and Bosphorus datasets, with a 93.30% recognition rate on Bosphorus using a combined codebook.

ABSTRACT

Facial expressions recognition (FER) of 3D face scans has received a significant amount of attention in recent years. Most of the facial expression recognition methods have been proposed using mainly 2D images. These methods suffer from several issues like illumination changes and pose variations. Moreover, 2D mapping from 3D images may lack some geometric and topological characteristics of the face. Hence, to overcome this problem, a multi-modal 2D + 3D feature-based method is proposed. We extract shallow features from the 3D images, and deep features using Convolutional Neural Networks (CNN) from the transformed 2D images. Combining these features into a compact representation uses covariance matrices as descriptors for both features instead of single-handedly descriptors. A covariance matrix learning is used as a manifold layer to reduce the deep covariance matrices size and enhance their discrimination power while preserving their manifold structure. We then use the Bag-of-Features (BoF) paradigm to quantize the covariance matrices after flattening. Accordingly, we obtained two codebooks using shallow and deep features. The global codebook is then used to feed an SVM classifier. High classification performances have been achieved on the BU-3DFE and Bosphorus datasets compared to the state-of-the-art methods.

Motivation & Objective

• To address the limitations of 2D-based facial expression recognition, such as sensitivity to illumination and pose variations, by leveraging 3D geometric data.
• To improve feature representation in 3D FER by combining deep features from CNNs and shallow geometric features from 3D scans.
• To enhance discriminative power and preserve manifold structure of feature descriptors using covariance matrix learning.
• To achieve robust classification by quantizing deep and shallow covariance matrices via the Bag-of-Features (BoF) paradigm.
• To outperform existing state-of-the-art methods on benchmark 3D facial expression datasets using a multi-modal 2D+3D feature fusion strategy.

Proposed method

• Extract shallow features directly from 3D face scans using geometric descriptors such as curvature and normal components.
• Generate deep features by applying pre-trained CNNs (VGG-16 and AlexNet) to 2D projections derived from 3D scans, such as depth and normal maps.
• Represent both shallow and deep features as covariance matrices to capture statistical relationships and preserve their Riemannian structure.
• Apply deep covariance matrix learning to reduce dimensionality, enhance discrimination, and preserve manifold properties of the covariance descriptors.
• Flatten the learned deep and shallow covariance matrices and apply the Bag-of-Features (BoF) paradigm to create separate codebooks for each modality.
• Combine the deep and shallow codebooks into a global feature vector and classify expressions using a multi-class SVM.

Experimental results

Research questions

• RQ1 Can combining deep and shallow features from 2D and 3D modalities improve 3D facial expression recognition performance?
• RQ2 Does covariance matrix representation of features enhance discrimination while preserving geometric structure in 3D FER?
• RQ3 Can deep covariance matrix learning effectively reduce dimensionality and improve feature compactness without losing discriminative power?
• RQ4 Does the BoF-based quantization of covariance matrices lead to better classification than single-modality or non-quantized approaches?
• RQ5 How does the proposed method compare to state-of-the-art 3D FER methods in terms of accuracy and robustness on standard benchmarks?

Key findings

• The proposed method achieved a 93.30% classification accuracy on the Bosphorus dataset, outperforming the state-of-the-art method by Ramya et al. (87.69%).
• The deep codebook alone achieved higher accuracy than the shallow codebook, indicating the superior discriminative power of deep features.
• The combination of deep and shallow codebooks yielded better performance than using the deep codebook alone, demonstrating complementary benefits of multi-modal feature fusion.
• Covariance matrix learning effectively reduced the size of deep covariance matrices while enhancing their discrimination power and preserving their Riemannian structure.
• The BoF paradigm proved effective in quantizing covariance descriptors, enabling efficient use of traditional classifiers like SVM in 3D FER.
• The method demonstrated strong generalization and robustness on both BU-3DFE and Bosphorus datasets, confirming its effectiveness for 3D facial expression recognition.