[论文解读] A Generalized Zero-Shot Framework for Emotion Recognition from Body Gestures

Jinting Wu, Yujia Zhang+2 more|arXiv (Cornell University)|Oct 13, 2020

Emotion and Mood Recognition参考文献 13被引用 7

一句话总结

该论文提出了一种用于从身体动作中识别情感的广义零样本学习（GZSL）框架，采用多分支架构，包含基于原型的检测器、流形正则化的堆叠自编码器以及情感分类器，所有分支共享一个带有自注意力机制的双向长短期记忆网络（BLSTM）特征提取器。该框架在MASR数据集上实现了最先进性能，相较于先前方法，未见类别准确率提升7.3%，调和平均值提升8.01%。

ABSTRACT

Although automatic emotion recognition from facial expressions and speech has made remarkable progress, emotion recognition from body gestures has not been thoroughly explored. People often use a variety of body language to express emotions, and it is difficult to enumerate all emotional body gestures and collect enough samples for each category. Therefore, recognizing new emotional body gestures is critical for better understanding human emotions. However, the existing methods fail to accurately determine which emotional state a new body gesture belongs to. In order to solve this problem, we introduce a Generalized Zero-Shot Learning (GZSL) framework, which consists of three branches to infer the emotional state of the new body gestures with only their semantic descriptions. The first branch is a Prototype-Based Detector (PBD) which is used to determine whether an sample belongs to a seen body gesture category and obtain the prediction results of the samples from the seen categories. The second branch is a Stacked AutoEncoder (StAE) with manifold regularization, which utilizes semantic representations to predict samples from unseen categories. Note that both of the above branches are for body gesture recognition. We further add an emotion classifier with a softmax layer as the third branch in order to better learn the feature representations for this emotion classification task. The input features for these three branches are learned by a shared feature extraction network, i.e., a Bidirectional Long Short-Term Memory Networks (BLSTM) with a self-attention module. We treat these three branches as subtasks and use multi-task learning strategies for joint training. The performance of our framework on an emotion recognition dataset is significantly superior to the traditional method of emotion classification and state-of-the-art zero-shot learning methods.

研究动机与目标

• 为解决在训练过程中未见过的新情感动作姿态的识别挑战，当前方法因数据稀缺与类别不一致而难以应对。
• 克服零样本学习模型在真实测试场景中对已见类别存在偏好、而忽略未见类别的偏差问题。
• 通过联合学习身体动作类别及其关联情感状态的语义描述，提升情感识别准确率。
• 开发一个统一框架，无需新类别的标注数据，即可泛化至已见与未见的情感动作姿态。

提出的方法

• 一个共享的双向长短期记忆网络（BLSTM）结合自注意力模块，从身体动作序列中提取深层时序特征。
• 基于原型的检测器（PBD）利用学习到的阈值，将样本分类为已见动作类别，并检测样本是否属于未见类别。
• 带有流形正则化的堆叠自编码器（StAE）利用语义嵌入学习重建视觉特征，从而实现对未见动作类别的分类。
• 带有Softmax层的情感分类器与主干网络联合训练，以增强情感分类的特征表示学习。
• 多任务学习同时优化三个分支，提升泛化能力，并平衡已见与未见类别上的性能表现。
• StAE中的流形正则化保留了视觉空间与语义空间的几何结构，从而提升对未见数据的泛化能力。

实验结果

研究问题

• RQ1 广义零样本学习框架能否有效识别训练数据中不存在的新情感动作姿态？
• RQ2 当测试数据同时包含已见与未见类别时，模型如何在两类上均保持高准确率？
• RQ3 引入情感分类器在多大程度上提升了基于身体动作的情感识别的泛化能力？
• RQ4 不同超参数对GZSL设置中重建、流形保持与分类性能之间的平衡有何影响？

主要发现

• 所提框架在已见与未见动作识别上的调和平均值（H）达到41.42%，相较先前最先进方法提升8.01%。
• 未见类别动作识别准确率（Accu）达到29.93%，相较次优方法绝对提升7.3%。
• 未见类别情感识别准确率（Accu^em）达到75.18%，相较先前最先进方法提升6.57%。
• 情感分类器分支的引入显著提升泛化性能，已见类别情感识别准确率（Accs^em）相较先前工作提升10.35%。
• 超参数分析表明，γ1、γ2与γ3在中间值时可实现最优性能，平衡重建、流形学习与推理时阈值设定。
• 该框架成功检测未见动作并将其映射至正确的情感状态，展现出在真实世界GZSL场景下的鲁棒性。