[Paper Review] Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network

Shervin Minaee, AmirAli Abdolrashidi|arXiv (Cornell University)|Feb 3, 2019

Emotion and Mood Recognition85 citations

TL;DR

Presents an attentional CNN with a spatial transformer to focus on salient face regions, achieving strong FER results across FER2013, CK+, JAFFE, and FERG datasets.

ABSTRACT

Facial expression recognition has been an active research area over the past few decades, and it is still challenging due to the high intra-class variation. Traditional approaches for this problem rely on hand-crafted features such as SIFT, HOG and LBP, followed by a classifier trained on a database of images or videos. Most of these works perform reasonably well on datasets of images captured in a controlled condition, but fail to perform as good on more challenging datasets with more image variation and partial faces. In recent years, several works proposed an end-to-end framework for facial expression recognition, using deep learning models. Despite the better performance of these works, there still seems to be a great room for improvement. In this work, we propose a deep learning approach based on attentional convolutional network, which is able to focus on important parts of the face, and achieves significant improvement over previous models on multiple datasets, including FER-2013, CK+, FERG, and JAFFE. We also use a visualization technique which is able to find important face regions for detecting different emotions, based on the classifier's output. Through experimental results, we show that different emotions seems to be sensitive to different parts of the face.

Motivation & Objective

• Address high intra-class variation in facial expression recognition (FER).
• Leverage attention to focus on salient facial regions for emotion detection.
• Demonstrate that a shallow, train-from-scratch network can outperform deeper models on FER tasks.
• Provide visualization of salient facial regions to interpret model decisions.

Proposed method

• End-to-end attentional convolutional network with four conv layers, max-pooling, ReLU, dropout, and two fully connected layers for feature extraction and classification.
• Incorporation of a spatial transformer network as a localization module to estimate an affine transformation T(θ) and warp the input to attend to important regions.
• Training via stochastic optimization (Adam) with a loss L_overall = L_classifier + λ||w(fc)||^2, plus data augmentation.
• Separate model trained per dataset to handle dataset-specific variations and limited data sizes.

Experimental results

Research questions

• RQ1 Does an attentional CNN outperform deeper networks on standard FER benchmarks?
• RQ2 Which facial regions are most salient for detecting different emotions according to the model?
• RQ3 How does the proposed method perform across FER2013, CK+, JAFFE, and FERG datasets?
• RQ4 What is the impact of training from scratch with attention on small FER datasets?

Key findings

• FER2013 test accuracy of about 70.02%.
• JAFFE dataset accuracy around 92.8%.
• CK+ dataset accuracy around 98.0%.
• FERG dataset accuracy around 99.3%.
• Visualization shows emotion-specific salient regions (e.g., mouth for happiness; eyes/brows for anger) and notes variability across images.