[论文解读] Cross-Domain Facial Expression Recognition: A Unified Evaluation Benchmark and Adversarial Graph Learning

Tianshui Chen, Tao Pu+4 more|arXiv (Cornell University)|Aug 3, 2020

Emotion and Mood Recognition参考文献 105被引用 12

一句话总结

本文提出了一套统一的跨域面部表情识别（CD-FER）评估基准，并引入了对抗性图表示自适应（AGRA）框架，该框架利用图卷积网络与对抗学习，联合适应跨域的整体与局部特征。AGRA在多个数据集上实现了最先进性能，在该基准上的平均准确率最高提升了8.5%。

ABSTRACT

To address the problem of data inconsistencies among different facial expression recognition (FER) datasets, many cross-domain FER methods (CD-FERs) have been extensively devised in recent years. Although each declares to achieve superior performance, fair comparisons are lacking due to the inconsistent choices of the source/target datasets and feature extractors. In this work, we first analyze the performance effect caused by these inconsistent choices, and then re-implement some well-performing CD-FER and recently published domain adaptation algorithms. We ensure that all these algorithms adopt the same source datasets and feature extractors for fair CD-FER evaluations. We find that most of the current leading algorithms use adversarial learning to learn holistic domain-invariant features to mitigate domain shifts. However, these algorithms ignore local features, which are more transferable across different datasets and carry more detailed content for fine-grained adaptation. To address these issues, we integrate graph representation propagation with adversarial learning for cross-domain holistic-local feature co-adaptation by developing a novel adversarial graph representation adaptation (AGRA) framework. Specifically, it first builds two graphs to correlate holistic and local regions within each domain and across different domains, respectively. Then, it extracts holistic-local features from the input image and uses learnable per-class statistical distributions to initialize the corresponding graph nodes. Finally, two stacked graph convolution networks (GCNs) are adopted to propagate holistic-local features within each domain to explore their interaction and across different domains for holistic-local feature co-adaptation. We conduct extensive and fair evaluations on several popular benchmarks and show that the proposed AGRA framework outperforms previous state-of-the-art methods.

研究动机与目标

• 为解决由于数据集和主干网络选择不一致，导致跨域面部表情识别（CD-FER）缺乏公平且一致的评估问题。
• 识别现有CD-FER方法在忽略更易跨域迁移的局部细粒度特征方面的局限性。
• 提出一个统一的评估基准，采用标准化的源/目标数据集与特征提取器，以实现公平比较。
• 开发一种新颖的AGRA框架，通过图表示学习与对抗训练，实现跨域整体与局部特征的联合自适应。

提出的方法

• 通过使用相同的源/目标数据集（以RAF-DB为源，CK+、JAFFE、SFEW2.0、FER2013、ExpW为目标）和ResNet-50主干网络，重新实现最先进方法，构建统一的CD-FER基准。
• 构建两个图：一个用于域内整体-局部特征相关性建模，另一个用于跨域特征对齐。
• 使用从输入图像中提取的整体与局部特征的可学习类别统计分布初始化图节点。
• 采用两个堆叠的图卷积网络（GCNs）：一个用于域内特征传播与交互，另一个用于跨域整体-局部特征协同自适应。
• 引入对抗学习，通过鼓励域不变的特征表示来最小化域偏移。
• 使用结合分类损失与对抗损失的多任务损失，联合优化特征学习与域对齐。

实验结果

研究问题

• RQ1 源/目标数据集与特征提取器的选择在多大程度上影响CD-FER评估的性能与公平性？
• RQ2 当前最先进CD-FER方法在多大程度上未能利用更易跨域迁移的局部细粒度特征？
• RQ3 通过基于图的消息传递实现整体与局部特征的联合自适应，是否能提升跨域面部表情识别性能？
• RQ4 在多样化数据集上，所提出的AGRA框架与现有方法相比，在准确率与鲁棒性方面表现如何？
• RQ5 将图网络集成到标准FER流程中，其计算成本与效率影响如何？

主要发现

• 所提出的AGRA框架在五个目标数据集（CK+、JAFFE、SFEW2.0、FER2013、ExpW）上实现了66.13%的平均准确率，平均优于先前最先进方法（ECAN）7.49%。
• 在CK+数据集上，AGRA达到85.27%的准确率，显著优于次佳方法（ECAN为79.77%）与基线（ICID为74.42%）。
• 消融实验表明，若移除对抗学习，平均准确率降至58.18%，证实其在学习域不变特征中的关键作用。
• 图网络的集成仅使推理时间增加2.54 ms，MACs增加0.00295%，表明计算开销极低。
• 在融合模块中用Transformer替代GCNs导致性能下降，证明图消息传递在整体-局部特征协同自适应中更具优势。
• 统一基准揭示，先前工作中数据集与主干网络选择不一致，导致性能比较产生误导，凸显了标准化评估的必要性。