[Paper Review] ARBEx: Attentive Feature Extraction with Reliability Balancing for Robust Facial Expression Learning

Azmine Toushik Wasi, Karlo Šerbetar+3 more|arXiv (Cornell University)|May 2, 2023

Emotion and Mood Recognition9 citations

TL;DR

ARBEx introduces an attentive feature extraction framework based on Window-Based Cross-Attention Vision Transformer with reliability balancing using learnable anchors and multi-head self-attention to handle bias, uncertainty, and poor class distributions in facial expression learning, achieving state-of-the-art results across datasets.

ABSTRACT

In this paper, we introduce a framework ARBEx, a novel attentive feature extraction framework driven by Vision Transformer with reliability balancing to cope against poor class distributions, bias, and uncertainty in the facial expression learning (FEL) task. We reinforce several data pre-processing and refinement methods along with a window-based cross-attention ViT to squeeze the best of the data. We also employ learnable anchor points in the embedding space with label distributions and multi-head self-attention mechanism to optimize performance against weak predictions with reliability balancing, which is a strategy that leverages anchor points, attention scores, and confidence values to enhance the resilience of label predictions. To ensure correct label classification and improve the models' discriminative power, we introduce anchor loss, which encourages large margins between anchor points. Additionally, the multi-head self-attention mechanism, which is also trainable, plays an integral role in identifying accurate labels. This approach provides critical elements for improving the reliability of predictions and has a substantial positive effect on final prediction capabilities. Our adaptive model can be integrated with any deep neural network to forestall challenges in various recognition tasks. Our strategy outperforms current state-of-the-art methodologies, according to extensive experiments conducted in a variety of contexts.

Motivation & Objective

• Address bias, uncertainty, and class distribution issues in facial expression learning (FEL).
• Develop a robust feature extraction pipeline leveraging Vision Transformer with cross-attention and multi-scale features.
• Introduce reliability balancing with anchor points and attentive corrections to stabilize label distributions.
• Provide a training pipeline with heavy augmentation and data refinement to mitigate overfitting on FEL data.

Proposed method

• Use a Window-Based Cross-Attention ViT (W-MCSA) to fuse multi-level features from image backbones and facial landmarks into a 768-d embedding, followed by a 128-d linear reduction for classification.
• Compute a primary label distribution l = softmax(f(e)) from the embedding e; estimate confidence via normalized entropy.
• Introduce reliability balancing with learnable anchors a^{i,j} in embedding space and multi-head self-attention to derive anchor-based correction t_g and attentive correction t_a.
• Combine corrections via a weighted scheme to produce final label distribution L_final, modulated by confidence-based weights.
• Train with a composite loss: class distribution loss, anchor loss (maximize margins between anchors), and center loss (pull embeddings toward class anchors).
• Adopt heavy data augmentation and a data refinement strategy to ensure balanced, diverse training batches across expressions.

Figure 1: A synopsis of ARBEx . Feature Extraction provides feature maps to generate initial predictions. Confidence distributions of initial labels are mostly inconsistent, unstable and unreliable. Reliability Balancing approach aids in stabilizing the distributions and addressing inconsistent and

Experimental results

Research questions

• RQ1 Can reliability balancing via anchors and attention improve FEL performance under biased/unbalanced label distributions?
• RQ2 Does window-based cross-attention ViT effectively fuse multi-scale features and landmarks for robust expression recognition?
• RQ3 How do anchor-based and attentive corrections influence label reliability and final prediction confidence?
• RQ4 What gains do heavy augmentation and data refinement provide in preventing overfitting on FEL benchmarks?

Key findings

• The ARBEx framework surpasses many state-of-the-art FEL systems across diverse datasets (Aff-Wild2, RAF-DB, JAFFE, FER-related benchmarks).
• Reliability balancing with learnable anchors and multi-head attention stabilizes label distributions and improves prediction confidence under label noise or bias.
• Window-based cross-attention ViT with multi-scale feature fusion yields robust embeddings suitable for effective facial expression classification.
• Experiments show the impact of anchor count K on accuracy, with diminishing returns beyond higher K values, indicating a sweet spot for anchors.
• The method integrates anchor loss, center loss, and class distribution loss to improve discriminability and stability of embeddings.

Figure 2: Pipeline of ARBEx . Heavy Augmentation is applied to the input images and Data Refinement method selects training batch with properly distributed classes for each epoch. Window-Based Cross-Attention ViT framework uses mutli-level feature extraction and integration to provide embeddings (Fe