[Paper Review] FER-YOLO-Mamba: Facial Expression Detection and Classification Based on Selective State Space

Hui Ma, Sen Lei+2 more|ArXiv.org|May 3, 2024

Face and Expression Recognition4 citations

TL;DR

This paper proposes FER-YOLO-Mamba, a novel YOLO-based model integrating Mamba's selective state space mechanisms to enhance facial expression detection and classification. By combining convolutional local feature extraction with Mamba’s long-range dependency modeling, the model achieves state-of-the-art performance on RAF-DB and SFEW datasets with improved mAP and F1 scores.

ABSTRACT

Facial Expression Recognition (FER) plays a pivotal role in understanding human emotional cues. However, traditional FER methods based on visual information have some limitations, such as preprocessing, feature extraction, and multi-stage classification procedures. These not only increase computational complexity but also require a significant amount of computing resources. Considering Convolutional Neural Network (CNN)-based FER schemes frequently prove inadequate in identifying the deep, long-distance dependencies embedded within facial expression images, and the Transformer's inherent quadratic computational complexity, this paper presents the FER-YOLO-Mamba model, which integrates the principles of Mamba and YOLO technologies to facilitate efficient coordination in facial expression image recognition and localization. Within the FER-YOLO-Mamba model, we further devise a FER-YOLO-VSS dual-branch module, which combines the inherent strengths of convolutional layers in local feature extraction with the exceptional capability of State Space Models (SSMs) in revealing long-distance dependencies. To the best of our knowledge, this is the first Vision Mamba model designed for facial expression detection and classification. To evaluate the performance of the proposed FER-YOLO-Mamba model, we conducted experiments on two benchmark datasets, RAF-DB and SFEW. The experimental results indicate that the FER-YOLO-Mamba model achieved better results compared to other models. The code is available from https://github.com/SwjtuMa/FER-YOLO-Mamba.

Motivation & Objective

• Address the limitations of traditional FER methods that rely on multi-stage pipelines involving manual preprocessing, feature extraction, and classification.
• Overcome the quadratic computational complexity of Transformers and the inability of CNNs to model long-range dependencies in facial expression data.
• Integrate the Mamba architecture—featuring linear complexity and selective state space mechanisms—into a YOLO-based object detection framework for end-to-end facial expression recognition.
• Develop a dual-branch FER-YOLO-VSS module that fuses local convolutional features with global contextual representations via a spatial attention mechanism.
• Demonstrate superior performance on benchmark datasets RAF-DB and SFEW, validating the model’s robustness and generalization in real-world scenarios.

Proposed method

• Propose FER-YOLO-Mamba, a YOLO-based object detection framework enhanced with a Vision Mamba backbone for joint facial expression detection and classification.
• Design the FER-YOLO-VSS dual-branch module, combining standard convolutional layers for local feature extraction with a selective state space model (SSM) for long-range dependency modeling.
• Integrate a spatial attention mechanism using global average pooling, multi-layer perceptrons (MLPs), and element-wise multiplication to emphasize discriminative regions in the feature maps.
• Utilize the Mamba architecture’s efficient state space transition dynamics to model sequential dependencies across spatial tokens with linear complexity, avoiding the quadratic cost of self-attention.
• Train the end-to-end model using cross-entropy loss for classification and standard YOLO loss for bounding box regression, enabling joint optimization of detection and recognition.
• Apply data augmentation and normalization techniques during training to improve robustness under varying lighting, pose, and occlusion conditions.

Experimental results

Research questions

• RQ1 Can a Vision Mamba-based architecture effectively model long-range dependencies in facial expression images while maintaining linear computational complexity?
• RQ2 How does the integration of a dual-branch FER-YOLO-VSS module improve feature representation compared to standalone CNN or Transformer-based models in FER tasks?
• RQ3 To what extent does the FER-YOLO-Mamba model outperform existing state-of-the-art models in terms of mAP, F1 score, and inference efficiency on RAF-DB and SFEW datasets?
• RQ4 Does the proposed model maintain robust performance under challenging conditions such as occlusion, poor lighting, and diverse head poses?
• RQ5 Can the spatial attention mechanism effectively highlight salient facial regions (e.g., eyes, mouth) to improve classification accuracy?

Key findings

• FER-YOLO-Mamba achieved an mAP of 80.31% on the RAF-DB dataset, outperforming YOLOvX, which achieved 78.40%.
• On the SFEW dataset, FER-YOLO-Mamba achieved an mAP of 66.7%, compared to 64.02% for YOLOvX, demonstrating improved generalization on challenging, real-world data.
• The model achieved an average F1 score of 0.75 on RAF-DB and 0.60 on SFEW, indicating strong performance across all emotion classes despite class imbalance.
• Visualization results on RAF-DB and SFEW showed that the model accurately localized facial regions and generated heatmaps highlighting key facial features, confirming effective attention to discriminative areas.
• The FER-YOLO-VSS dual-branch module enhanced feature representation by selectively amplifying critical regions, as evidenced by the improved precision and recall in classes like 'Happy' and 'Surprise'.
• The model demonstrated robustness in complex scenes with background interference, maintaining high confidence scores and accurate bounding box predictions.