[Paper Review] MVT: Mask Vision Transformer for Facial Expression Recognition in the wild

Hanting Li, Mingzhe Sui+3 more|arXiv (Cornell University)|Jun 8, 2021

Emotion and Mood Recognition48 citations

TL;DR

The paper introduces a pure transformer-based Mask Vision Transformer (MVT) for FER in the wild, featuring a mask generation network (MGN) and a dynamic relabeling module, achieving state-of-the-art results on RAF-DB, FERPlus, and AffectNet-7 (with comparable results on AffectNet-8).

ABSTRACT

Facial Expression Recognition (FER) in the wild is an extremely challenging task in computer vision due to variant backgrounds, low-quality facial images, and the subjectiveness of annotators. These uncertainties make it difficult for neural networks to learn robust features on limited-scale datasets. Moreover, the networks can be easily distributed by the above factors and perform incorrect decisions. Recently, vision transformer (ViT) and data-efficient image transformers (DeiT) present their significant performance in traditional classification tasks. The self-attention mechanism makes transformers obtain a global receptive field in the first layer which dramatically enhances the feature extraction capability. In this work, we first propose a novel pure transformer-based mask vision transformer (MVT) for FER in the wild, which consists of two modules: a transformer-based mask generation network (MGN) to generate a mask that can filter out complex backgrounds and occlusion of face images, and a dynamic relabeling module to rectify incorrect labels in FER datasets in the wild. Extensive experimental results demonstrate that our MVT outperforms state-of-the-art methods on RAF-DB with 88.62%, FERPlus with 89.22%, and AffectNet-7 with 64.57%, respectively, and achieves a comparable result on AffectNet-8 with 61.40%.

Motivation & Objective

• Address facial expression recognition in the wild with challenging backgrounds, occlusions, and annotation uncertainties.
• Propose a pure transformer-based FER framework (MVT) that uses masking to filter background noise.
• Introduce a dynamic relabeling strategy to rectify mislabeled samples during training.
• Leverage pretrained Vision Transformers to achieve robust FER performance across diverse wild datasets.

Proposed method

• Introduce a mask generation network (MGN) based on transformers to produce per-image masks that suppress background and occlusions.
• Feed masked images into a Vision Transformer (ViT/DeiT-S backbone) for expression classification using a class token.
• Train MGN with a GAN-like setup using a transformer discriminator and a novel generator loss that encourages selective masking via variance of predicted expressions and a target mask area.
• Apply a dynamic relabeling module that decides label relabeling with a threshold f(P_gt) + δ, where the threshold increases with the given label probability P_gt.
• Fine-tune the classifier (DeiT-S) on masked inputs, keeping MGN frozen during FER training.
• ] ,
• research_questions
• :[

Experimental results

Research questions

• RQ1 Can a pure transformer architecture effectively handle FER in the wild when augmented with learned masks?
• RQ2 Does a dynamic relabeling strategy improve training stability and accuracy on wild FER datasets?
• RQ3 How does masking affect robustness to backgrounds, occlusion, and pose variations across RAF-DB, FERPlus, and AffectNet?

Key findings

• MVT achieves 88.62% on RAF-DB, 89.22% on FERPlus, and 64.57% on AffectNet-7, with 61.40% on AffectNet-8.
• The Mask Generation Network (MGN) effectively filters background and occlusion, improving accuracy compared to baselines.
• Dynamic relabeling outperforms fixed-threshold relabeling and stabilizes training across datasets.
• The approach yields state-of-the-art results on RAF-DB, FERPlus, AffectNet-7 and competitive results on AffectNet-8.
• Ablation shows mask ratio and dynamic relabeling function choices impact performance, with dataset-specific optimal settings (e.g., m=15% for RAF-DB, m=20% for FERPlus).
• The method is robust to occlusion and pose variants, outperforming baselines on Occlusion-RAF-DB and achieving competitive results on Pose-RAF-DB.