[论文解读] Audio-visual Representation Learning for Anomaly Events Detection in Crowds

Junyu Gao, Maoguo Gong+1 more|arXiv (Cornell University)|Oct 28, 2021

Anomaly Detection Techniques and Applications参考文献 55被引用 6

一句话总结

本文提出音频-视觉表征学习（AVRL），一种多模态框架，通过融合基于3D CNN的视频特征与从对数梅尔频谱图编码的VGGish音频特征，提升人群场景中的异常检测性能。该方法在SHADE数据集上达到最先进性能，尤其在低能见度条件下显著提升准确率，证明了音频在检测枪击、击倒等事件中的互补作用。

ABSTRACT

In recent years, anomaly events detection in crowd scenes attracts many researchers' attention, because of its importance to public safety. Existing methods usually exploit visual information to analyze whether any abnormal events have occurred due to only visual sensors are generally equipped in public places. However, when an abnormal event in crowds occurs, sound information may be discriminative to assist the crowd analysis system to determine whether there is an abnormality. Compare with vision information that is easily occluded, audio signals have a certain degree of penetration. Thus, this paper attempt to exploit multi-modal learning for modeling the audio and visual signals simultaneously. To be specific, we design a two-branch network to model different types of information. The first is a typical 3D CNN model to extract temporal appearance features from video clips. The second is an audio CNN for encoding Log Mel-Spectrogram of audio signals. Finally, by fusing the above features, a more accurate prediction will be produced. We conduct the experiments on SHADE dataset, a synthetic audio-visual dataset in surveillance scenes, and find introducing audio signals effectively improves the performance of anomaly events detection and outperforms other state-of-the-art methods. Furthermore, we will release the code and the pre-trained models as soon as possible.

研究动机与目标

• 解决仅依赖视觉的异常检测在人群场景中的局限性，其中遮挡和低能见度会阻碍检测效果。
• 探究音频信号作为视觉数据的补充模态，在检测异常事件中的潜力。
• 开发一种有效的多模态学习框架，融合视觉与音频表征以提升检测准确率。
• 证明在视觉线索退化时，音频特征在低能见度场景下的有效性。
• 基于同步音视频的合成数据集，为监控应用中的音视频表征学习建立基线。

提出的方法

• 采用3D ResNet主干网络从视频片段中提取时空特征，捕捉运动与外观动态。
• 将原始音频信号转换为对数梅尔频谱图，并通过VGGish模型编码，实现判别性音频表征学习。
• 通过拼接方式在特征层面融合视觉与音频特征，实现多模态线索的联合建模。
• 使用交叉熵损失函数，端到端训练AVRL框架，用于多类别异常事件分类。
• 通过随机时间与空间补丁进行数据增强，评估模型鲁棒性及各模态的贡献。
• 开展消融实验，比较仅视觉（3D ResNet）、仅音频（VGGish）与音视频联合（AVRL）配置的性能。

Figure 1: Some typical samples in SHADE dataset. The green box contains three groups of image sequences of Shoot, Knockdown, and Arrest, of which red boxes in frames highlight the incident location. The magenta box visualizes the audio signals and Mel-Spectrograms for the above crowd scenes.

实验结果

研究问题

• RQ1 当视觉数据被遮挡或退化时，音频信号是否能提升人群场景中的异常检测性能？
• RQ2 视觉与音频表征在检测特定异常事件（如枪击或打斗）时，各自贡献有何不同？
• RQ3 在多模态异常检测中，结合3D-CNN与音频CNN特征的最优融合策略是什么？
• RQ4 AVRL在不同光照条件下（尤其是视觉质量低的黑暗场景）的性能表现如何？
• RQ5 空间扰动与时间扰动在多大程度上影响视觉与音频表征的鲁棒性？

主要发现

• AVRL在'枪击'事件上达到95.9%的top-1准确率，显著优于仅视觉的3D ResNet（89.8%）与仅音频的VGGish（91.8%）。
• 在黑暗子集（视觉质量差）中，AVRL在'打斗'、'奔跑'与'追逐'事件上的检测准确率优于仅视觉方法。
• 仅音频的VGGish在'枪击'事件上达到91.8%的高准确率，表明特定音频模式对某些异常事件具有高度判别性。
• 消融实验证实，视觉特征平均贡献更大，但音频特征能提升所有类别性能。
• 空间补丁对准确率的负面影响强于时间补丁，表明空间上下文对事件识别比时间连续性更为关键。
• 当空间补丁比例达80%时，仅视觉模型在部分类别上的性能低于仅音频模型，表明过度视觉退化可能误导模型。

Figure 2: AVRL for abnormal events detection in crowd scenes. The details of 3D ResNet and VGGish is the same as [ 28 ] and [ 29 ] respectively.