[Paper Review] Multi-scale multi-modal micro-expression recognition algorithm based on transformer

Fengping Wang, Jie Li+3 more|arXiv (Cornell University)|Jan 8, 2023

Advanced Computing and Algorithms4 citations

TL;DR

This paper proposes a transformer-based multi-scale, multi-modal micro-expression recognition algorithm that fuses motion and texture features across multiple spatial scales using multi-head self-attention and cross-modal contrastive learning. It achieves state-of-the-art performance with 78.73% accuracy on SMIC and 0.9071 F1-score on CASMEII combined dataset.

ABSTRACT

A micro-expression is a spontaneous unconscious facial muscle movement that can reveal the true emotions people attempt to hide. Although manual methods have made good progress and deep learning is gaining prominence. Due to the short duration of micro-expression and different scales of expressed in facial regions, existing algorithms cannot extract multi-modal multi-scale facial region features while taking into account contextual information to learn underlying features. Therefore, in order to solve the above problems, a multi-modal multi-scale algorithm based on transformer network is proposed in this paper, aiming to fully learn local multi-grained features of micro-expressions through two modal features of micro-expressions - motion features and texture features. To obtain local area features of the face at different scales, we learned patch features at different scales for both modalities, and then fused multi-layer multi-headed attention weights to obtain effective features by weighting the patch features, and combined cross-modal contrastive learning for model optimization. We conducted comprehensive experiments on three spontaneous datasets, and the results show the accuracy of the proposed algorithm in single measurement SMIC database is up to 78.73% and the F1 value on CASMEII of the combined database is up to 0.9071, which is at the leading level.

Motivation & Objective

• To address the challenge of recognizing micro-expressions due to their short duration and varying spatial scales in facial regions.
• To improve feature extraction by capturing multi-grained local features across multiple scales in both motion and texture modalities.
• To enhance representation learning by fusing multi-scale features using multi-head self-attention mechanisms.
• To optimize the model using cross-modal contrastive learning to align motion and texture embeddings.
• To achieve state-of-the-art performance on spontaneous micro-expression recognition benchmarks.

Proposed method

• The method employs a dual-stream transformer architecture to process motion and texture features separately at multiple scales.
• It extracts patch-level features from facial regions at different resolutions for both modalities to capture multi-scale local patterns.
• Multi-layer multi-headed self-attention is applied to each modality to weight and aggregate patch features based on contextual relationships.
• The attention weights from different scales are fused to generate enhanced, scale-robust representations.
• Cross-modal contrastive learning is used to align the motion and texture feature spaces, improving generalization and representation quality.
• The final features are combined and classified using a classifier head trained end-to-end.

Experimental results

Research questions

• RQ1 Can a multi-scale, multi-modal approach improve micro-expression recognition by capturing fine-grained spatial patterns across different facial regions?
• RQ2 How effectively can a transformer-based architecture model long-range dependencies and contextual relationships in micro-expression sequences?
• RQ3 To what extent does cross-modal contrastive learning enhance feature alignment between motion and texture modalities?
• RQ4 Does the proposed method outperform existing state-of-the-art models on spontaneous micro-expression datasets?
• RQ5 How does the integration of multi-scale patch features affect recognition accuracy and robustness?

Key findings

• The proposed method achieved 78.73% accuracy on the SMIC spontaneous micro-expression dataset, outperforming prior methods.
• On the combined CASMEII dataset, the model reached an F1-score of 0.9071, demonstrating strong performance in multi-class recognition.
• The use of multi-scale patch features significantly improved the model's ability to capture subtle facial movements across varying spatial resolutions.
• Cross-modal contrastive learning contributed to better alignment of motion and texture embeddings, enhancing feature discriminability.
• The fusion of multi-head self-attention across scales led to more robust and context-aware feature representations.
• The model demonstrated state-of-the-art performance across multiple evaluation metrics on benchmark datasets.