[Paper Review] MFEViT: A Robust Lightweight Transformer-based Network for Multimodal 2D+3D Facial Expression Recognition

Hanting Li, Mingzhe Sui+2 more|arXiv (Cornell University)|Sep 20, 2021

Face and Expression Recognition31 references4 citations

TL;DR

This paper proposes MFEViT, a lightweight, pure transformer-based model for multimodal 2D+3D facial expression recognition that uses an alternative fusion strategy to merge RGB and depth features before the transformer encoder, and a sample filtering module to reassign noisy samples to subclasses. It achieves state-of-the-art accuracy of 90.83% on BU-3DFE and 90.28% on Bosphorus with only ~23M parameters, outperforming multi-branch CNNs.

ABSTRACT

Vision transformer (ViT) has been widely applied in many areas due to its self-attention mechanism that help obtain the global receptive field since the first layer. It even achieves surprising performance exceeding CNN in some vision tasks. However, there exists an issue when leveraging vision transformer into 2D+3D facial expression recognition (FER), i.e., ViT training needs mass data. Nonetheless, the number of samples in public 2D+3D FER datasets is far from sufficient for evaluation. How to utilize the ViT pre-trained on RGB images to handle 2D+3D data becomes a challenge. To solve this problem, we propose a robust lightweight pure transformer-based network for multimodal 2D+3D FER, namely MFEViT. For narrowing the gap between RGB and multimodal data, we devise an alternative fusion strategy, which replaces each of the three channels of an RGB image with the depth-map channel and fuses them before feeding them into the transformer encoder. Moreover, the designed sample filtering module adds several subclasses for each expression and move the noisy samples to their corresponding subclasses, thus eliminating their disturbance on the network during the training stage. Extensive experiments demonstrate that our MFEViT outperforms state-of-the-art approaches with an accuracy of 90.83% on BU-3DFE and 90.28% on Bosphorus. Specifically, the proposed MFEViT is a lightweight model, requiring much fewer parameters than multi-branch CNNs. To the best of our knowledge, this is the first work to introduce vision transformer into multimodal 2D+3D FER. The source code of our MFEViT will be publicly available online.

Motivation & Objective

• To address the challenge of limited 2D+3D facial expression recognition (FER) datasets by adapting vision transformers pre-trained on large-scale RGB data.
• To reduce model complexity and parameter count compared to existing multi-branch CNNs used in multimodal FER.
• To mitigate the negative impact of noisy samples and large intra-class variations in facial expression data during training.
• To introduce a pure transformer-based architecture into multimodal 2D+3D FER for the first time, leveraging self-attention for global feature learning.

Proposed method

• Proposes an alternative fusion (AF) strategy that replaces each of the three RGB channels with the depth-map channel and fuses them before feeding into the vision transformer encoder, narrowing the domain gap between RGB and multimodal data.
• Designs a sample filtering (SF) module that creates subclasses for each expression and reassigns noisy samples to these subclasses during training to reduce their interference on the main class.
• Applies the vision transformer (ViT) encoder on fused multimodal features after data-level fusion, enabling global receptive field learning from the first layer.
• Performs 2D+3D feature extraction by first projecting 3D scans into aligned RGB and depth-map pairs, followed by surface processing to enhance data quality.
• Uses a single-branch transformer architecture, eliminating the need for multi-branch CNNs and significantly reducing model parameters.
• At inference, merges all subclasses back into their original main classes for final classification and accuracy evaluation.

Experimental results

Research questions

• RQ1 Can a vision transformer pre-trained on large-scale RGB data be effectively adapted to low-resource 2D+3D facial expression recognition?
• RQ2 How can the domain gap between RGB and multimodal (2D+3D) data be minimized to enable effective transfer learning?
• RQ3 Can a single-branch transformer-based model outperform multi-branch CNNs in 2D+3D FER while using significantly fewer parameters?
• RQ4 To what extent does filtering noisy samples into subclasses improve model generalization and accuracy in facial expression recognition?

Key findings

• MFEViT achieves a state-of-the-art accuracy of 90.83% on the BU-3DFE dataset and 90.28% on the Bosphorus dataset, surpassing existing SOTA methods.
• The model reduces parameter count to approximately 23 million, significantly fewer than multi-branch CNNs such as VGG-M-DF (327M) and DA-CNN (463M).
• The sample filtering module improves accuracy by 1.39% on BU-3DFE and 1.11% on Bosphorus compared to a baseline without subclass filtering.
• The alternative fusion strategy effectively narrows the distribution gap between RGB and multimodal data, enabling better transfer from ImageNet-pretrained ViT.
• Visualization confirms that the distance between normal samples and noisy samples is significantly reduced after subclass reassignment, validating the effectiveness of the filtering module.
• Confusion matrices show higher accuracy for expressions like happiness and surprise, while fear and sadness remain more challenging due to subtle muscle movements.