[论文解读] Knowing What to Listen to: Early Attention for Deep Speech Representation Learning.

Amirhossein Hajavi, Ali Etemad|arXiv (Cornell University)|Sep 3, 2020

Speech Recognition and Synthesis参考文献 34被引用 7

一句话总结

本文提出了一种新型注意力机制——细粒度早期频域注意力（FEFA），使深度学习模型能够聚焦于语音信号中的单个频带。通过将FEFA集成到CNN主干网络中，该模型在VoxCeleb和IEMOCAP数据集上的说话人识别任务中实现了最先进性能，并在噪声环境下表现出更强的鲁棒性。

ABSTRACT

Deep learning techniques have considerably improved speech processing in recent years. Speech representations extracted by deep learning models are being used in a wide range of tasks such as speech recognition, speaker recognition, and speech emotion recognition. Attention models play an important role in improving deep learning models. However current attention mechanisms are unable to attend to fine-grained information items. In this paper we propose the novel Fine-grained Early Frequency Attention (FEFA) for speech signals. This model is capable of focusing on information items as small as frequency bins. We evaluate the proposed model on two popular tasks of speaker recognition and speech emotion recognition. Two widely used public datasets, VoxCeleb and IEMOCAP, are used for our experiments. The model is implemented on top of several prominent deep models as backbone networks to evaluate its impact on performance compared to the original networks and other related work. Our experiments show that by adding FEFA to different CNN architectures, performance is consistently improved by substantial margins, even setting a new state-of-the-art for the speaker recognition task. We also tested our model against different levels of added noise showing improvements in robustness and less sensitivity compared to the backbone networks.

研究动机与目标

• 解决现有注意力机制在聚焦语音信号中细粒度频带级特征方面的局限性。
• 提升深度学习模型在说话人识别和语音情绪识别等语音表征学习任务中的性能。
• 通过实现对频谱分量的早期、细粒度注意力，增强模型对噪声的鲁棒性。
• 在多种CNN架构和基准数据集上评估FEFA，证明其性能提升的一致性。

提出的方法

• FEFA旨在在网络早期阶段对单个频带进行注意力聚焦，从而实现对频谱细节的精细关注。
• 该注意力机制被设计为轻量级模块，集成于CNN主干网络的输入层或早期卷积层。
• 通过从输入谱图中提取的查询、键和值投影，计算频带上的注意力权重。
• 该机制具有可微性，支持端到端训练，可与主干网络联合优化。
• 在多种架构（如ResNet、DenseNet）上评估FEFA，以检验其泛化能力与性能增益。
• 在VoxCeleb和IEMOCAP上进行模型训练与测试，并开展噪声鲁棒性的消融研究。

实验结果

研究问题

• RQ1 注意力机制能否被调整以聚焦于单个频带，而非粗粒度的时间或通道级特征？
• RQ2 对频谱分量进行早期、细粒度的注意力是否能提升说话人与情绪识别任务的性能？
• RQ3 FEFA在不同水平背景噪声下如何增强模型的鲁棒性？
• RQ4 FEFA能否在多种CNN架构和数据集上持续提升性能？

主要发现

• FEFA在多个CNN架构上均持续提升了说话人识别与语音情绪识别任务的性能。
• 该模型在VoxCeleb说话人识别基准上达到了新的最先进性能。
• FEFA在噪声环境下表现出更强的鲁棒性，相较于主干网络，性能下降更小。
• 即使应用于强基线模型，FEFA的集成也能带来显著的性能提升。