[论文解读] ARBEx: Attentive Feature Extraction with Reliability Balancing for Robust Facial Expression Learning

Azmine Toushik Wasi, Karlo Šerbetar+3 more|arXiv (Cornell University)|May 2, 2023

Emotion and Mood Recognition被引用 9

一句话总结

ARBEx 引入基于 Window-Based Cross-Attention Vision Transformer 的注意力特征提取框架，结合可学习锚点的可靠性平衡和多头自注意力，以应对面部表情学习中的偏差、不确定性和分布不均等问题，在多个数据集上实现了最先进的结果。

ABSTRACT

In this paper, we introduce a framework ARBEx, a novel attentive feature extraction framework driven by Vision Transformer with reliability balancing to cope against poor class distributions, bias, and uncertainty in the facial expression learning (FEL) task. We reinforce several data pre-processing and refinement methods along with a window-based cross-attention ViT to squeeze the best of the data. We also employ learnable anchor points in the embedding space with label distributions and multi-head self-attention mechanism to optimize performance against weak predictions with reliability balancing, which is a strategy that leverages anchor points, attention scores, and confidence values to enhance the resilience of label predictions. To ensure correct label classification and improve the models' discriminative power, we introduce anchor loss, which encourages large margins between anchor points. Additionally, the multi-head self-attention mechanism, which is also trainable, plays an integral role in identifying accurate labels. This approach provides critical elements for improving the reliability of predictions and has a substantial positive effect on final prediction capabilities. Our adaptive model can be integrated with any deep neural network to forestall challenges in various recognition tasks. Our strategy outperforms current state-of-the-art methodologies, according to extensive experiments conducted in a variety of contexts.

研究动机与目标

• 解决面部表情学习（FEL）中的偏差、不确定性和类别分布问题。
• 开发基于跨注意力和多尺度特征的 Vision Transformer 的鲁棒特征提取管线。
• 引入带有锚点的可靠性平衡和注意力修正以稳定标签分布。
• 提供含强增强和数据精炼的训练管线，降低 FEL 数据的过拟合。

提出的方法

• 使用 Window-Based Cross-Attention ViT (W-MCSA) 将图像骨干和面部关键点的多层特征融合为 768-d 的嵌入，再经 128-d 线性降维用于分类。
• 从嵌入 e 计算主标签分布 l = softmax(f(e))；通过归一化熵估计置信度。
• 引入带可学习锚点 a^{i,j} 的嵌入空间可靠性平衡和多头自注意力，得到基于锚点的校正 t_g 和注意力校正 t_a。
• 通过加权方案将校正结合，生成最终标签分布 L_final，并以置信度为权重进行调制。
• 以综合损失训练：类别分布损失、锚点损失（最大化锚点之间的边距）以及中心损失（将嵌入拉向类别锚点）。
• 采用强数据增强和数据精炼策略，确保跨表情的训练批次平衡且多样。

Figure 1: A synopsis of ARBEx . Feature Extraction provides feature maps to generate initial predictions. Confidence distributions of initial labels are mostly inconsistent, unstable and unreliable. Reliability Balancing approach aids in stabilizing the distributions and addressing inconsistent and

实验结果

研究问题

• RQ1 通过锚点和注意力的可靠性平衡，是否能在标注分布偏置/不平衡的条件下提升 FEL 性能？
• RQ2 基于窗口的跨注意力 ViT 是否能有效融合多尺度特征和关键点以实现鲁棒表情识别？
• RQ3 锚点基校正和注意力修正如何影响标签的可靠性及最终预测置信度？
• RQ4 强增强和数据精炼在防止 FEL 基准测试过拟合方面带来哪些提升？

主要发现

• ARBEx 框架在多个数据集（Aff-Wild2、RAF-DB、JAFFE、FER 相关基准）上超过了许多先进的 FEL 系统。
• 带可学习锚点的可靠性平衡与多头注意力稳定标签分布，并在存在标签噪声或偏差时提升预测置信度。
• 基于窗口的跨注意力 ViT 与多尺度特征融合，生成鲁棒嵌入，适合表情分类。
• 实验显示锚点数量 K 对准确率有影响，超过较高的 K 值后收益递减，存在锚点的最佳取值点。
• 该方法整合锚点损失、中心损失与类别分布损失，提升嵌入的可辨别性和稳定性。

Figure 2: Pipeline of ARBEx . Heavy Augmentation is applied to the input images and Data Refinement method selects training batch with properly distributed classes for each epoch. Window-Based Cross-Attention ViT framework uses mutli-level feature extraction and integration to provide embeddings (Fe