[论文解读] Feature Pyramid Attention based Residual Neural Network for Environmental Sound Classification

Liguang Zhou, Yuhongze Zhou+4 more|arXiv (Cornell University)|May 28, 2022

Music and Audio Processing被引用 6

一句话总结

本文提出了一种特征金字塔注意力网络（FPAM），通过在频谱图中显式定位语义相关的时频区域，提升了环境声音分类的性能。通过在多尺度特征上集成金字塔空间注意力与通道注意力模块，FPAM在ESC-50和ESC-10数据集上分别取得了91.6%和99.3%的准确率，达到当前最优水平，同时通过注意力可视化有效过滤了无关噪声和静音帧。

ABSTRACT

Environmental sound classification (ESC) is a challenging problem due to the unstructured spatial-temporal relations that exist in the sound signals. Recently, many studies have focused on abstracting features from convolutional neural networks while the learning of semantically relevant frames of sound signals has been overlooked. To this end, we present an end-to-end framework, namely feature pyramid attention network (FPAM), focusing on abstracting the semantically relevant features for ESC. We first extract the feature maps of the preprocessed spectrogram of the sound waveform by a backbone network. Then, to build multi-scale hierarchical features of sound spectrograms, we construct a feature pyramid representation of the sound spectrograms by aggregating the feature maps from multi-scale layers, where the temporal frames and spatial locations of semantically relevant frames are localized by FPAM. Specifically, the multiple features are first processed by a dimension alignment module. Afterward, the pyramid spatial attention module (PSA) is attached to localize the important frequency regions spatially with a spatial attention module (SAM). Last, the processed feature maps are refined by a pyramid channel attention (PCA) to localize the important temporal frames. To justify the effectiveness of the proposed FPAM, visualization of attention maps on the spectrograms has been presented. The visualization results show that FPAM can focus more on the semantic relevant regions while neglecting the noises. The effectiveness of the proposed methods is validated on two widely used ESC datasets: the ESC-50 and ESC-10 datasets. The experimental results show that the FPAM yields comparable performance to state-of-the-art methods. A substantial performance increase has been achieved by FPAM compared with the baseline methods.

研究动机与目标

• 为解决由于环境声音具有非结构化的时空模式且缺乏先验语义结构而导致的分类挑战。
• 通过显式聚焦于语义相关的帧而非同等对待所有区域，改进基于CNN模型的特征学习。
• 将多尺度层次化特征与空间注意力及通道注意力机制相结合，以更好地表征显著的声学模式。
• 通过可视化和在标准ESC数据集上的基准评估，验证所提出注意力机制的有效性。

提出的方法

• 该方法首先使用主干CNN从音频波形的对数梅尔频谱图中提取多尺度特征图。
• 通过维度对齐模块在融合前对不同尺度的特征进行归一化。
• 金字塔空间注意力（PSA）模块利用特征图上的空间注意力机制，定位重要的频带区域。
• 金字塔通道注意力（PCA）模块通过在不同尺度上突出相关的时间帧来优化特征。
• 将联合注意力模块以层次化方式应用，以强调判别性区域，同时抑制噪声和静音帧。
• 该框架采用交叉熵损失端到端训练，并通过mix-up数据增强技术提升泛化能力。

实验结果

研究问题

• RQ1 多尺度注意力机制是否能够提升环境声音频谱图中语义相关帧的定位能力？
• RQ2 将金字塔空间注意力与通道注意力相结合，与标准CNN相比，如何增强ESC任务的特征表征能力？
• RQ3 所提出的FPAM在多大程度上减少了对频谱图中噪声或静音帧的依赖？
• RQ4 该注意力机制是否能提升在缺乏先验结构的多样化环境声音类别上的泛化能力和鲁棒性？

主要发现

• FPAM在ESC-50数据集上达到91.6%的准确率，显著优于基线方法（86.2%），并达到最先进性能。
• 在ESC-10数据集上，FPAM达到99.3%的准确率，表明其在更小且更平衡的数据集上具有强大的泛化能力。
• 注意力图可视化结果表明，FPAM能有效突出显著的、语义相关的信号频带，同时过滤掉静音和无关帧。
• 消融实验显示，将mix-up数据增强与FPAM结合可进一步提升性能，在ESC-50上达到90.5%，在ESC-10上达到98.5%。
• 训练曲线显示损失持续下降、准确率稳步上升，表明优化过程稳定且高效。
• 混淆矩阵显示，误分类主要发生在相似的父类别之间（如天气声音），表明模型学习到了有意义的语义区分。