[Paper Review] PAtt-Lite: Lightweight Patch and Attention MobileNet for Challenging Facial Expression Recognition

Jia Le Ngwe, Kian Ming Lim+2 more|arXiv (Cornell University)|Jun 16, 2023

Emotion and Mood Recognition49 references4 citations

TL;DR

PAtt-Lite proposes a lightweight MobileNetV1-based model with a patch extraction block and self-attention classifier to improve facial expression recognition (FER) under challenging conditions. By focusing on local facial features and enhancing feature representation through attention, it achieves state-of-the-art accuracy on CK+, RAF-DB, FER2013, FERPlus, and their challenging subsets, outperforming prior methods with only 1.10M parameters.

ABSTRACT

Facial Expression Recognition (FER) is a machine learning problem that deals with recognizing human facial expressions. While existing work has achieved performance improvements in recent years, FER in the wild and under challenging conditions remains a challenge. In this paper, a lightweight patch and attention network based on MobileNetV1, referred to as PAtt-Lite, is proposed to improve FER performance under challenging conditions. A truncated ImageNet-pre-trained MobileNetV1 is utilized as the backbone feature extractor of the proposed method. In place of the truncated layers is a patch extraction block that is proposed for extracting significant local facial features to enhance the representation from MobileNetV1, especially under challenging conditions. An attention classifier is also proposed to improve the learning of these patched feature maps from the extremely lightweight feature extractor. The experimental results on public benchmark databases proved the effectiveness of the proposed method. PAtt-Lite achieved state-of-the-art results on CK+, RAF-DB, FER2013, FERPlus, and the challenging conditions subsets for RAF-DB and FERPlus.

Motivation & Objective

• Address the challenge of accurate facial expression recognition (FER) in real-world, in-the-wild conditions such as occlusion and pose variations.
• Overcome class imbalance issues, particularly for rare negative expressions like contempt and fear, which are underrepresented in existing datasets.
• Develop a lightweight FER model that maintains high accuracy without relying on large, resource-intensive neural networks.
• Improve feature representation in MobileNetV1 by introducing localized patch extraction and attention-based classification.

Proposed method

• A truncated ImageNet-pretrained MobileNetV1 is used as the backbone feature extractor to reduce model size and computational cost.
• A lightweight patch extraction block splits the feature maps from MobileNetV1 into four non-overlapping spatial regions to emphasize significant local facial features.
• A self-attention classifier is inserted after global average pooling, using a dot-product attention mechanism between fully connected layers to better model long-range dependencies in the feature maps.
• The patch extraction block and attention classifier are integrated into a single lightweight architecture to enhance representation learning under challenging conditions.
• The model is trained end-to-end on public FER benchmarks including CK+, RAF-DB, FER2013, and FERPlus, with fine-tuning on challenging condition subsets.
• The method is evaluated on standard benchmarks and challenging subsets (occlusion, pose >30°, pose >45°) to assess robustness.

Experimental results

Research questions

• RQ1 Can a lightweight patch extraction mechanism improve FER performance under challenging conditions such as occlusion and extreme poses?
• RQ2 Does integrating a self-attention classifier into a lightweight MobileNetV1 backbone enhance feature representation and classification accuracy in FER?
• RQ3 How does PAtt-Lite compare to state-of-the-art methods in terms of accuracy and parameter efficiency on both standard and challenging FER benchmarks?
• RQ4 Can the proposed method effectively mitigate performance degradation on low-resource expression classes like contempt and fear despite class imbalance?

Key findings

• PAtt-Lite achieved state-of-the-art accuracy on the CK+ dataset with 98.12% top-1 accuracy, surpassing previous methods.
• On RAF-DB, PAtt-Lite achieved 92.23% accuracy on the occlusion subset, 95.35% on Pose >30°, and 94.44% on Pose >45°, outperforming the previous SOTA (Facial Chirality) by over 3% with significantly fewer parameters.
• On FERPlus, PAtt-Lite achieved 93.22% on occlusion, 96.07% on Pose >30°, and 92.58% on Pose >45°, marking the first method to exceed 90% accuracy on all three challenging subsets.
• PAtt-Lite outperformed RAN [5] by 6.39% in overall accuracy on FERPlus while using 10 times fewer parameters (1.10M vs. 11.2M).
• The model achieved over 95% accuracy on all but the most imbalanced classes (Contempt, Disgust, Fear) on RAF-DB and FER2013, with performance drops expected due to class imbalance.
• The ablation study confirmed that both the patch extraction block and attention classifier contribute significantly to performance gains, especially under challenging conditions.