[论文解读] Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models

Sarala Padi, Seyed Omid Sadjadi+2 more|arXiv (Cornell University)|Feb 16, 2022

Emotion and Mood Recognition被引用 10

一句话总结

本文提出了一种多模态情感识别框架，利用预训练的ResNet模型进行语音特征提取，并对文本使用微调后的BERT模型，通过晚期融合策略结合两者以提升性能。尽管面临数据稀缺的挑战，该方法在IEMOCAP数据集上取得了最先进（SOTA）的结果，文本模态的单模态准确率为70.33%，多模态融合的准确率为71.12%。

ABSTRACT

Automatic emotion recognition plays a key role in computer-human interaction as it has the potential to enrich the next-generation artificial intelligence with emotional intelligence. It finds applications in customer and/or representative behavior analysis in call centers, gaming, personal assistants, and social robots, to mention a few. Therefore, there has been an increasing demand to develop robust automatic methods to analyze and recognize the various emotions. In this paper, we propose a neural network-based emotion recognition framework that uses a late fusion of transfer-learned and fine-tuned models from speech and text modalities. More specifically, we i) adapt a residual network (ResNet) based model trained on a large-scale speaker recognition task using transfer learning along with a spectrogram augmentation approach to recognize emotions from speech, and ii) use a fine-tuned bidirectional encoder representations from transformers (BERT) based model to represent and recognize emotions from the text. The proposed system then combines the ResNet and BERT-based model scores using a late fusion strategy to further improve the emotion recognition performance. The proposed multimodal solution addresses the data scarcity limitation in emotion recognition using transfer learning, data augmentation, and fine-tuning, thereby improving the generalization performance of the emotion recognition models. We evaluate the effectiveness of our proposed multimodal approach on the interactive emotional dyadic motion capture (IEMOCAP) dataset. Experimental results indicate that both audio and text-based models improve the emotion recognition performance and that the proposed multimodal solution achieves state-of-the-art results on the IEMOCAP benchmark.

研究动机与目标

• 通过利用预训练模型的迁移学习，解决语音和文本情感识别中的数据稀缺问题。
• 通过梅尔频谱图增强和微调技术，提升情感识别的泛化能力。
• 通过融合互补的音频与文本表征，提升模型性能。
• 通过从说话人识别和BERT迁移学习，在IEMOCAP基准上实现最先进结果。
• 证明将预训练于数据丰富任务（如说话人识别、语言建模）的模型复用于低资源情感识别任务的有效性。

提出的方法

• 对原始在说话人识别任务上预训练的ResNet34模型进行微调，用于语音情感识别，输入为高分辨率对数梅尔频谱图。
• 通过随机时间-频率掩码实施频谱图增强，以增加训练数据多样性并减少过拟合。
• 使用预训练的BERT模型从转录文本中提取上下文相关的句子嵌入，并对情感分类任务进行微调。
• 通过晚期融合策略结合音频和文本模型的预测情感得分，以利用模态间的互补性。
• 在包含脚本化和即兴表达的完整IEMOCAP数据集上，采用5折留一说话人交叉验证策略进行训练与评估。
• 通过平均两种模态的置信度得分来优化融合策略，以提升最终预测性能。

实验结果

研究问题

• RQ1 从预训练的说话人识别模型迁移学习是否能在数据稀缺条件下提升语音情感识别性能？
• RQ2 在文本数据上微调基于BERT的模型是否能显著提升情感识别准确率？
• RQ3 音频与文本预测的晚期融合是否优于单模态方法？
• RQ4 所提方法在IEMOCAP基准上与最先进模型相比表现如何？
• RQ5 频谱图增强在语音情感识别中在多大程度上提升了泛化能力？

主要发现

• 所提的仅音频模型在IEMOCAP数据集上达到65.97%的话语级准确率，优于wav2vec-Base和HuBERT-Base等先前方法。
• 仅文本模型达到70.33%的话语级准确率，超过Bert+CNN和BERT+attention等先前SOTA方法。
• 晚期融合的多模态系统达到71.12%的话语级准确率，为IEMOCAP基准的最先进结果。
• 频谱图增强显著提升了模型泛化能力并减少了过拟合，尤其在标注数据有限的情况下。
• 对BERT模型进行情感识别微调相比将其作为固定特征提取器，带来了显著的性能提升。
• 尽管参数量少于2200万，该模型在参数效率方面优于大型自监督模型如HuBERT-Large和WavLM-Large。