[Paper Review] Urban Sound Classification : striving towards a fair comparison

Augustin Arnault, Baptiste Hanssens+1 more|arXiv (Cornell University)|Oct 22, 2020

Music and Audio Processing25 references4 citations

TL;DR

This paper presents a DCASE 2020 Task 5-winning solution for urban sound classification that achieves a macro-AUPRC of 0.82 (coarse) and 0.62 (fine) on validation data, with 89.7% and 85.41% accuracy on ESC-50 and US8k, respectively. It introduces a fair comparison framework using consistent input representation, metrics, and optimizer to enable reproducible evaluation of models across datasets.

ABSTRACT

Urban sound classification has been achieving remarkable progress and is still an active research area in audio pattern recognition. In particular, it allows to monitor the noise pollution, which becomes a growing concern for large cities. The contribution of this paper is two-fold. First, we present our DCASE 2020 task 5 winning solution which aims at helping the monitoring of urban noise pollution. It achieves a macro-AUPRC of 0.82 / 0.62 for the coarse / fine classification on validation set. Moreover, it reaches accuracies of 89.7% and 85.41% respectively on ESC-50 and US8k datasets. Second, it is not easy to find a fair comparison and to reproduce the performance of existing models. Sometimes authors copy-pasting the results of the original papers which is not helping reproducibility. As a result, we provide a fair comparison by using the same input representation, metrics and optimizer to assess performances. We preserve data augmentation used by the original papers. We hope this framework could help evaluate new architectures in this field. For better reproducibility, the code is available on our GitHub repository.

Motivation & Objective

• To present a state-of-the-art solution for urban sound classification that won the DCASE 2020 Task 5 challenge.
• To address the lack of fair and reproducible model comparison in audio pattern recognition by standardizing input representation, metrics, and optimization.
• To evaluate model performance across multiple datasets (ESC-50, US8k, SONYC-UST) using a unified framework to ensure objective comparison.
• To improve reproducibility by releasing code and highlighting inconsistencies in spectrogram computation and metric implementation.
• To investigate the impact of label quality and relabeling strategies on model performance, especially on noisy datasets like SONYC-UST.

Proposed method

• The model uses raw audio input converted into log-mel spectrograms using Librosa to ensure consistent feature extraction.
• A hybrid architecture combines a global feature extractor based on modified TALNet with 10 convolutional blocks and a Bi-GRU layer, and a specific feature extractor with residual connections.
• The model employs multi-head self-attention to capture long-range temporal dependencies, improving performance on fine-grained classification.
• Data augmentation techniques include time warping, frequency masking, time masking, and random resizing, adapted from SpecAugment and Argus.
• A relabeling strategy was applied to SONYC-UST to correct inconsistent labels from volunteer annotations, improving model robustness.
• All models are evaluated using the same optimizer (Adam), loss function (binary cross-entropy), and metrics (macro-AUPRC, accuracy), ensuring fair comparison.

Experimental results

Research questions

• RQ1 How does a unified evaluation framework with consistent input representation, metrics, and optimizer improve the reproducibility and fairness of model comparison in urban sound classification?
• RQ2 What impact does label quality have on model performance, particularly on datasets like SONYC-UST with potentially unreliable volunteer annotations?
• RQ3 Can a self-attention-enhanced architecture outperform convolutional models in capturing long-range temporal patterns in urban sound events?
• RQ4 How do different spectrogram computation methods (e.g., Librosa vs. PyTorch) affect model performance and reproducibility?
• RQ5 To what extent do variations in data augmentation techniques and implementation details (e.g., Mixup variants) affect model generalization and reported results?

Key findings

• The proposed model achieved a macro-AUPRC of 0.82 on coarse-level and 0.62 on fine-level classification for the DCASE 2020 Task 5 validation set.
• On the ESC-50 dataset, the model reached 89.7% accuracy, outperforming several baseline models.
• On the US8k dataset, the model achieved 85.41% accuracy, demonstrating strong generalization across datasets.
• The model outperformed CNN10 on SONYC-UST for coarse-level classification, but CNN10 surpassed it on fine-level tasks.
• The relabeling strategy significantly improved performance on SONYC-UST, indicating that label noise is a major confounder in real-world datasets.
• Discrepancies in metric computation—especially AUC and F1—were observed across frameworks, highlighting the need for standardized evaluation.