[Paper Review] Facial Emotion Recognition using Convolutional Neural Networks

Akash Saravanan, Gurudutt Perichetla+1 more|arXiv (Cornell University)|Oct 12, 2019

Emotion and Mood RecognitionPsychology10 references17 citations

TL;DR

This paper proposes a six-convolutional-layer Convolutional Neural Network (CNN) for real-time facial emotion recognition, achieving 60% accuracy on the FER-2013 dataset using only a single dataset and a lightweight architecture. The model leverages spatial feature extraction via filters and max pooling, with top-2 predictions offering improved reliability, demonstrating near-state-of-the-art performance with minimal data and real-time inference capability.

ABSTRACT

Facial expression recognition is a topic of great interest in most fields from artificial intelligence and gaming to marketing and healthcare. The goal of this paper is to classify images of human faces into one of seven basic emotions. A number of different models were experimented with, including decision trees and neural networks before arriving at a final Convolutional Neural Network (CNN) model. CNNs work better for image recognition tasks since they are able to capture spacial features of the inputs due to their large number of filters. The proposed model consists of six convolutional layers, two max pooling layers and two fully connected layers. Upon tuning of the various hyperparameters, this model achieved a final accuracy of 0.60.

Motivation & Objective

• To develop a real-time facial emotion recognition system capable of classifying human faces into seven emotions: anger, disgust, fear, happiness, sadness, surprise, and neutral.
• To address challenges in real-world deployment such as lighting variations, partial face visibility, and imbalanced training data for rare emotions like disgust and fear.
• To achieve high accuracy using only the FER-2013 dataset without combining multiple datasets, emphasizing model efficiency and generalization.
• To evaluate the model’s performance on live webcam input to validate real-time applicability in practical domains like healthcare, gaming, and marketing.
• To explore the impact of hyperparameter tuning and model architecture on recognition accuracy, particularly for low-resource emotion classes.

Proposed method

• A six-convolutional-layer CNN with two max pooling layers and two fully connected layers was designed to extract hierarchical spatial features from facial images.
• The model was trained using the Adam optimizer with tuned hyperparameters to improve convergence and generalization on the FER-2013 dataset.
• Emotion classification was performed by selecting the emotion with the highest predicted probability from the output softmax layer.
• A live testing module was implemented using webcam input, enabling real-time inference with no perceptible delay.
• Confusion matrix analysis and top-2 prediction evaluation were used to assess model reliability and identify misclassification patterns.
• Data augmentation was considered as a future improvement path to address class imbalance, particularly for underrepresented emotions like disgust and fear.

Experimental results

Research questions

• RQ1 Can a single, lightweight CNN architecture achieve near-state-of-the-art facial emotion recognition accuracy using only the FER-2013 dataset?
• RQ2 How does the model perform on real-time webcam input, and what is its inference speed and reliability in uncontrolled environments?
• RQ3 To what extent do top-2 predicted emotions improve overall recognition accuracy compared to top-1 predictions?
• RQ4 Why does the model struggle with certain emotions like disgust and fear, and how does class imbalance affect performance?
• RQ5 Can hyperparameter tuning and architectural choices significantly improve performance without relying on multi-dataset training?

Key findings

• The proposed CNN achieved a final accuracy of 60% on the FER-2013 dataset, demonstrating strong performance with a single dataset and minimal architectural complexity.
• Happiness and surprise were the most accurately predicted emotions, with accuracy approaching 60%, while disgust was predicted correctly in slightly over half of the cases.
• The model showed high reliability in real-time webcam testing, correctly identifying happiness and surprise in nearly all instances and neutral/sad expressions about half the time.
• Anger and fear were frequently confused with each other, and disgust was rarely predicted correctly, indicating challenges in distinguishing subtle or low-frequency emotions.
• The top-2 prediction strategy significantly improved reliability, as the correct emotion was often the second most likely prediction, confirming its utility for robust inference.
• The model achieved real-time inference with no noticeable delay, indicating strong potential for deployment in live applications such as gaming, healthcare, and marketing.