[Paper Review] Enriched Long-term Recurrent Convolutional Network for Facial Micro-Expression Recognition

Huai-Qian Khor, John See+2 more|arXiv (Cornell University)|May 22, 2018

Emotion and Mood Recognition15 references17 citations

TL;DR

This paper proposes an Enriched Long-term Recurrent Convolutional Network (ELRCN) for facial micro-expression recognition, combining CNN-based spatial feature extraction with LSTM-based temporal modeling. The framework uses two variants—spatial enrichment via channel-wise input stacking and temporal enrichment via feature-wise stacking—achieving strong performance without data augmentation, with Grad-CAM visualizations confirming attention on AU-relevant facial regions.

ABSTRACT

Facial micro-expression (ME) recognition has posed a huge challenge to researchers for its subtlety in motion and limited databases. Recently, handcrafted techniques have achieved superior performance in micro-expression recognition but at the cost of domain specificity and cumbersome parametric tunings. In this paper, we propose an Enriched Long-term Recurrent Convolutional Network (ELRCN) that first encodes each micro-expression frame into a feature vector through CNN module(s), then predicts the micro-expression by passing the feature vector through a Long Short-term Memory (LSTM) module. The framework contains two different network variants: (1) Channel-wise stacking of input data for spatial enrichment, (2) Feature-wise stacking of features for temporal enrichment. We demonstrate that the proposed approach is able to achieve reasonably good performance, without data augmentation. In addition, we also present ablation studies conducted on the framework and visualizations of what CNN "sees" when predicting the micro-expression classes.

Motivation & Objective

• Address the challenge of recognizing subtle, brief facial micro-expressions due to their low visibility and limited training data.
• Overcome limitations of handcrafted features, which require domain-specific tuning and are not generalizable.
• Develop a deep learning framework that effectively captures spatio-temporal dynamics in micro-expressions using end-to-end learning.
• Investigate the impact of different feature representations (raw pixels vs. optical flow) and network architectures on recognition performance.
• Provide visual interpretability of model predictions using Grad-CAM to validate attention on action unit (AU)-relevant facial regions.

Proposed method

• Use a pre-trained VGG-16 CNN to extract deep spatial features from each micro-expression frame, with feature maps from the last and second-last fully connected layers tested.
• Pass the extracted spatial features through a Long Short-Term Memory (LSTM) network to model temporal dependencies across frames.
• Implement two network variants: (1) Channel-wise stacking of input data (RGB, optical flow) for spatial enrichment, and (2) Feature-wise stacking of CNN-embedded features for temporal enrichment.
• Employ optical flow as input to enhance motion sensitivity, replacing raw pixel intensities for improved discrimination of subtle facial dynamics.
• Apply gradient-weighted class activation mapping (Grad-CAM) on the last convolutional layer to visualize which facial regions contribute most to classification decisions.
• Conduct ablation studies comparing different LSTM architectures (single vs. two-layer), feature extraction layers, and input modalities (pixel vs. flow).

Experimental results

Research questions

• RQ1 Can a deep learning-based approach outperform handcrafted features in micro-expression recognition without requiring extensive hyperparameter tuning?
• RQ2 How does the choice of input modality (raw pixels vs. optical flow) affect recognition performance in micro-expression tasks?
• RQ3 Does spatial enrichment via input channel stacking or temporal enrichment via feature stacking yield better performance on single-database and cross-database evaluations?
• RQ4 To what extent do model predictions align with expert-annotated action units (AUs), as validated by Grad-CAM visualizations?
• RQ5 What is the optimal configuration of the LSTM component (e.g., number of layers, number of units) for micro-expression recognition using deep features?

Key findings

• The ELRCN with temporal enrichment (TE) outperforms the spatial enrichment (SE) variant on single-database evaluation, achieving higher recognition accuracy.
• The spatial enrichment (SE) variant generalizes better to cross-database settings, indicating improved domain transfer capability.
• Optical flow features consistently outperform raw pixel intensities as input, demonstrating greater sensitivity to subtle facial motion in micro-expressions.
• A single-layer LSTM performs better than a two-layer LSTM when using deep CNN features, suggesting that deeper recurrence may not improve performance and increases computational cost.
• Features from the last fully connected layer (4096-dim) of VGG-16 yield the most discriminative representations for micro-expression classification.
• Grad-CAM visualizations confirm that the model attends to AU-relevant facial regions—such as the lip corner puller (AU12) and eyebrow raisers (AU1)—matching expert annotations.