[Paper Review] Ambient Sound Helps: Audiovisual Crowd Counting in Extreme Conditions

Di Hu, Lichao Mou+5 more|ArXiv.org|May 14, 2020

Video Surveillance and Tracking Methods51 references30 citations

TL;DR

This paper introduces audiovisual crowd counting and a large DISCO benchmark, showing ambient sound improves crowd counting under low illumination, noise, occlusion, and low resolution.

ABSTRACT

Visual crowd counting has been recently studied as a way to enable people counting in crowd scenes from images. Albeit successful, vision-based crowd counting approaches could fail to capture informative features in extreme conditions, e.g., imaging at night and occlusion. In this work, we introduce a novel task of audiovisual crowd counting, in which visual and auditory information are integrated for counting purposes. We collect a large-scale benchmark, named auDiovISual Crowd cOunting (DISCO) dataset, consisting of 1,935 images and the corresponding audio clips, and 170,270 annotated instances. In order to fuse the two modalities, we make use of a linear feature-wise fusion module that carries out an affine transformation on visual and auditory features. Finally, we conduct extensive experiments using the proposed dataset and approach. Experimental results show that introducing auditory information can benefit crowd counting under different illumination, noise, and occlusion conditions. The dataset and code will be released. Code and data have been made available

Motivation & Objective

• Motivate a novel audiovisual crowd counting task inspired by human multisensory perception.
• Create a large-scale DISCO dataset with 1,935 image-audio clips and 170,270 annotated head instances.
• Develop an audiovisual counting model with feature-wise fusion to leverage ambient sound for counting.
• Evaluate how audio cues affect counting across illumination, noise, occlusion, and resolution conditions.

Proposed method

• Use the first ten layers of VGG16 as visual feature extractor.
• Represent audio with Log Mel-Spectrograms and a 6-layer CNN (VGGish-derived).
• Fuse audio and visual features via a linear, feature-wise fusion block that yields gamma and beta parameters for affine transformation derived from audio via average pooling and FC layers.
• Stack multiple fusion blocks (6 blocks) with dilated convolutions to increase receptive field and produce full-resolution density maps.
• Train with L2 loss between predicted and ground-truth density maps.
• Compare AudioCSRNet against vision-only baselines (MCNN, CSRNet, SANet, CANNet) using MAE and MSE.

Experimental results

Research questions

• RQ1 Does combining visual and auditory features improve crowd counting over vision-only models in extreme conditions?
• RQ2 How do audiovisual counting results vary under different illumination, noise, and occlusion conditions?
• RQ3 What is an effective and computationally efficient fusion mechanism for audiovisual cues in crowd counting?

Key findings

• Audio-augmented models consistently outperform vision-only baselines on low-quality images (low illumination, noise, occlusion, low resolution).
• AudioCSRNet achieves lower MAE and MSE than CSRNet under challenging conditions, and remains competitive on high-quality images.
• Audio information enables counting even when visual information is absent or severely degraded (e.g., R=0 in illumination experiment).
• Joint audiovisual models show robustness across varying illumination and occlusion levels, with larger performance gaps as visual quality deteriorates.
• Using CNN-based audio representations (spectrogram-based) yields better results than MFCC+LSTM; AudioCANNet also benefits CANNet in low-quality regimes.