[Paper Review] Learning Vision Transformer with Squeeze and Excitation for Facial Expression Recognition

Mouath Aouayeb, Wassim Hamidouche+3 more|arXiv (Cornell University)|Jul 7, 2021

Emotion and Mood Recognition44 citations

TL;DR

This work integrates a Squeeze and Excitation block into a Vision Transformer for facial expression recognition, with FER-2013 pretraining, achieving state-of-the-art single-model results on CK+ and SFEW and competitive results on JAFFE and RAF-DB.

ABSTRACT

As various databases of facial expressions have been made accessible over the last few decades, the Facial Expression Recognition (FER) task has gotten a lot of interest. The multiple sources of the available databases raised several challenges for facial recognition task. These challenges are usually addressed by Convolution Neural Network (CNN) architectures. Different from CNN models, a Transformer model based on attention mechanism has been presented recently to address vision tasks. One of the major issue with Transformers is the need of a large data for training, while most FER databases are limited compared to other vision applications. Therefore, we propose in this paper to learn a vision Transformer jointly with a Squeeze and Excitation (SE) block for FER task. The proposed method is evaluated on different publicly available FER databases including CK+, JAFFE,RAF-DB and SFEW. Experiments demonstrate that our model outperforms state-of-the-art methods on CK+ and SFEW and achieves competitive results on JAFFE and RAF-DB.

Motivation & Objective

• Address facial expression recognition (FER) with data-limited datasets by enhancing a Vision Transformer (ViT) with a lightweight SE block.
• Fine-tune a ViT-B16-224 on FER databases (CK+, JAFFE, RAF-DB, SFEW) and assess performance against state-of-the-art methods.
• Analyze how SE affects the ViT's attention and robustness, including attention visualizations and cross-database evaluation.
• Evaluate cross-database generalization (e.g., CK+ cross-database evaluation) to gauge transferability of the proposed model.

Proposed method

• Use a ViT-B16-224 base pretrained on JFT-300M and fine-tuned on ImageNet-1K for FER tasks.
• Incorporate a Squeeze and Excitation (SE) block on top of the ViT classification token to recalibrate channel-wise responses via two fully connected layers with a sigmoid gate (Excitation); apply element-wise multiplication with the cls token.
• Train with data augmentation (Random Horizontal Flip, Random Grayscale, varying brightness/contrast/saturation), Cutout and Mixup regularization, and AdamW optimizer (lr = 1.6e-4, batch size 16).
• Evaluate four databases (CK+, JAFFE, RAF-DB, SFEW) and perform ablation studies comparing ViT, ViT+SE, with/without FER-2013 pretraining.
• Visualize learned representations with t-SNE and attention maps (Grad-CAM, Score-CAM, Eigen-CAM) to analyze local vs global attention and SE impact.

Experimental results

Research questions

• RQ1 Can ViT with an SE block overcome FER data scarcity and deliver strong performance on both lab-controlled and wild FER datasets?
• RQ2 What is the effect of the SE block on ViT’s attention and the learned representations for FER?
• RQ3 Does pretraining on FER-2013 synergize with SE to improve FER performance, especially on smaller or more variable datasets?
• RQ4 How well does the proposed model generalize across different FER databases (cross-database evaluation)?

Key findings

• ViT+SE achieves 99.80% accuracy on CK+ with 10-fold cross-validation, exceeding prior methods.
• On JAFFE, ViT+SE reaches 92.92% accuracy (10-fold cross-validation).
• On RAF-DB, ViT+SE attains 87.22% accuracy, competitive with state-of-the-art single-model results.
• On SFEW, ViT+SE achieves 54.29% accuracy (single model), with higher ensemble-based results reported elsewhere.
• FER-2013 pretraining plus SE yields notable gains on wild datasets (RAF-DB, SFEW) and generally enhances robustness and separability (as seen in t-SNE and attention maps).
• Cross-database CK+ evaluation shows ViT+SE generalizes better than ResNet50, with SE providing the best cross-database transfer from CK+ to other datasets; however, overall generalization remains modest.