[论文解读] Relation-guided acoustic scene classification aided with event embeddings

Yuanbo Hou, Bo Soo Kang+2 more|arXiv (Cornell University)|May 1, 2022

Music and Audio Processing被引用 5

一句话总结

该论文提出了一种关系引导的声学场景分类（RGASC）模型，通过利用伪标签音频事件和学习到的场景-事件关系矩阵，提升场景分类的准确性。通过在TUT2018数据集上联合训练双塔CNN，RGASC利用场景中事件出现频率的先验知识，引导细粒度事件线索与粗粒度场景标签的融合，在地铁站类别上实现17.38%的准确率提升，在行人街道类别上实现13.36%的提升。

ABSTRACT

In real life, acoustic scenes and audio events are naturally correlated. Humans instinctively rely on fine-grained audio events as well as the overall sound characteristics to distinguish diverse acoustic scenes. Yet, most previous approaches treat acoustic scene classification (ASC) and audio event classification (AEC) as two independent tasks. A few studies on scene and event joint classification either use synthetic audio datasets that hardly match the real world, or simply use the multi-task framework to perform two tasks at the same time. Neither of these two ways makes full use of the implicit and inherent relation between fine-grained events and coarse-grained scenes. To this end, this paper proposes a relation-guided ASC (RGASC) model to further exploit and coordinate the scene-event relation for the mutual benefit of scene and event recognition. The TUT Urban Acoustic Scenes 2018 dataset (TUT2018) is annotated with pseudo labels of events by a simple and efficient audio-related pre-trained model PANN, which is one of the state-of-the-art AEC models. Then, a prior scene-event relation matrix is defined as the average probability of the presence of each event type in each scene class. Finally, the two-tower RGASC model is jointly trained on the real-life dataset TUT2018 for both scene and event classification. The following results are achieved. 1) RGASC effectively coordinates the true information of coarse-grained scenes and the pseudo information of fine-grained events. 2) The event embeddings learned from pseudo labels under the guidance of prior scene-event relations help reduce the confusion between similar acoustic scenes. 3) Compared with other (non-ensemble) methods, RGASC improves the scene classification accuracy on the real-life dataset.

研究动机与目标

• 解决缺乏同时包含声学场景和事件注释的真实世界数据集的问题。
• 克服将声学场景分类与事件分类视为独立任务的局限性。
• 利用场景与事件之间隐含的、真实世界的关系，以提升分类性能。
• 证明即使事件的伪标签未经验证，只要通过场景-事件关系引导，也能增强场景分类性能。
• 开发一种联合学习框架，实现场景与事件识别之间的双向知识迁移。

提出的方法

• 使用预训练的PANN模型在TUT2018数据集上为527个音频事件类别生成伪标签。
• 构建先验场景-事件关系矩阵 $ R_{SE} $，表示每类场景下事件的平均概率。
• 设计一种双塔CNN架构，采用共享特征提取和由 $ R_{SE} $ 引导的交叉注意力融合机制。
• 通过端到端的多任务学习方法，同时训练场景分类与事件分类任务。
• 使用t-SNE可视化分析嵌入空间的聚类效果与混淆程度的降低。
• 应用知识蒸馏原理，将PANN作为教师模型，AEC塔作为学生模型，实现事件表征知识的迁移。

实验结果

研究问题

• RQ1 当通过真实世界场景-事件关系引导时，伪标签音频事件能否提升声学场景分类性能？
• RQ2 引入学习到的场景-事件关系矩阵在多大程度上提升了对相似声学场景的判别能力？
• RQ3 未经验证的伪标签在多大程度上仍能对场景分类性能的提升产生贡献？
• RQ4 具有关系引导融合机制的双塔模型是否优于标准多任务学习或单任务基线模型？
• RQ5 关系引导的融合是否能减少声学相似场景（如公交车与有轨电车，交通与公园）之间的混淆？

主要发现

• 与基线puASC相比，RGASC在地铁站类别上实现17.38%的准确率提升，在行人街道类别上实现13.36%的提升。
• 模型在交通类别上达到90.24%的准确率，在机场类别上达到86.42%，优于基线puASC模型。
• t-SNE可视化显示，RGASC嵌入空间中相似场景（如公交车/有轨电车、广场/公园）的聚类更加清晰，表明混淆程度降低。
• 即使标签并非真实标签，场景-事件关系矩阵仍能有效引导伪标签事件信息的融合。
• AEC塔从伪标签中学习到有意义的事件表征，从而增强场景分类器对相似场景的区分能力。
• 该方法表明，通过场景-事件关系引导，将预训练模型（PANN）的知识蒸馏到学生AEC塔中，可显著提升场景分类性能。