[Paper Review] Facial Expression Recognition with Swin Transformer

Jun-Hwa Kim, Nam‐Ho Kim+1 more|arXiv (Cornell University)|Mar 25, 2022

Emotion and Mood Recognition19 citations

TL;DR

This paper proposes a three-stream Swin Transformer-based model for facial expression recognition on the in-the-wild Aff-Wild2 dataset, leveraging visual, temporal, and audio modalities. It introduces half-mix jittering, a novel data augmentation technique that improves generalization on imbalanced data, achieving a 35.71 F1-score on the validation set, outperforming the baseline by over 12 points.

ABSTRACT

The task of recognizing human facial expressions plays a vital role in various human-related systems, including health care and medical fields. With the recent success of deep learning and the accessibility of a large amount of annotated data, facial expression recognition research has been mature enough to be utilized in real-world scenarios with audio-visual datasets. In this paper, we introduce Swin transformer-based facial expression approach for an in-the-wild audio-visual dataset of the Aff-Wild2 Expression dataset. Specifically, we employ a three-stream network (i.e., Visual stream, Temporal stream, and Audio stream) for the audio-visual videos to fuse the multi-modal information into facial expression recognition. Experimental results on the Aff-Wild2 dataset show the effectiveness of our proposed multi-modal approaches.

Motivation & Objective

• To improve facial expression recognition performance on the in-the-wild Aff-Wild2 dataset using multi-modal data.
• To address class imbalance in facial expression datasets through a novel data augmentation strategy.
• To leverage the representational power of Swin Transformers for visual, temporal, and audio streams in a unified multi-modal framework.
• To validate the effectiveness of half-mix jittering in enhancing model robustness on partial facial inputs.

Proposed method

• A three-stream network architecture is employed, with separate Swin Transformer encoders for visual, temporal, and audio streams.
• The visual stream processes single cropped face images using a Swin Transformer Large backbone.
• The temporal stream uses S3D-convolutional layers to extract spatiotemporal features from 16-frame clips sampled every second.
• The audio stream converts audio signals into mel-spectrograms, which are then processed by a Swin Transformer encoder.
• Half-mix jittering is applied during training, where a portion of the face (left/right or top/bottom) is replaced with a reference face from the same batch, using a learnable mask and label mixing.
• Score fusion is performed at inference by averaging the predictions from all three streams.

Experimental results

Research questions

• RQ1 Can a Swin Transformer-based multi-modal model effectively recognize facial expressions in unconstrained, in-the-wild video data?
• RQ2 How does half-mix jittering data augmentation improve performance on imbalanced facial expression datasets?
• RQ3 What is the contribution of audio and temporal features to facial expression recognition beyond static visual inputs?
• RQ4 Does the proposed three-stream architecture with late fusion outperform single-stream or early fusion baselines?

Key findings

• The proposed three-stream model achieved a validation F1-score of 35.71, significantly outperforming the baseline model, which scored 23.00.
• The visual stream alone achieved a high F1-score of 34.74, demonstrating the strong performance of Swin Transformer on single-frame facial expression recognition.
• The addition of temporal and audio streams improved performance from 35.08 (visual + temporal) to 35.71 (full three-stream fusion), indicating complementary information from all modalities.
• Half-mix jittering contributed to improved generalization, particularly benefiting minority classes such as Anger, Disgust, Fear, and Surprise, which were used as reference images during augmentation.
• The model showed robustness to partial facial inputs, as half-mix jittering explicitly encourages attention to salient facial regions.