[Paper Review] CRNNs for Urban Sound Tagging with spatiotemporal context

Augustin Arnault, Nicolas Riche|arXiv (Cornell University)|Aug 24, 2020

Music and Audio Processing18 references4 citations

TL;DR

This paper proposes a CRNN-based model integrating log-mel spectrograms, spatiotemporal metadata (time, location), and hybrid audio embeddings (generic and specific) for hierarchical multilabel urban sound tagging. By combining a TALNet-inspired CNN-Transformer for specific embeddings, pre-trained generic embeddings, and Time2Vec-embedded metadata, the model achieves state-of-the-art performance on the DCASE 2020 Task 5 benchmark, with a macro-AUPRC of 0.8107 on coarse tags and 0.7040 on fine tags, surpassing the baseline in both metrics.

ABSTRACT

This paper describes CRNNs we used to participate in Task 5 of the DCASE 2020 challenge. This task focuses on hierarchical multilabel urban sound tagging with spatiotemporal context. The code is available on our GitHub repository at https://github.com/multitel-ai/urban-sound-tagging.

Motivation & Objective

• To address the challenge of hierarchical multilabel urban sound tagging with spatiotemporal context in real-world audio recordings.
• To improve performance on the DCASE 2020 Task 5 benchmark by integrating diverse input modalities: spectrograms, metadata, and audio embeddings.
• To investigate the effectiveness of combining generic pre-trained audio embeddings with task-specific learned embeddings in multilabel audio tagging.
• To evaluate the impact of advanced training techniques and data augmentation on model generalization and robustness.
• To develop a scalable and modular deep learning framework for urban sound event detection and classification.

Proposed method

• The model uses log-mel spectrograms (64 bands, 0–8 kHz, Hanning window, hop length 1103) as the primary audio input representation.
• Spatiotemporal metadata (hour, day, week, location) are embedded using Time2Vec (T2V) followed by a transformer encoder to capture periodic and non-periodic patterns.
• Specific audio embeddings are generated via a TALNet-inspired architecture with Group Normalization, Weight Standardization, and a transformer encoder replacing bi-GRU layers.
• Generic audio embeddings are derived from a pre-trained TALNet model fine-tuned on AudioSet, providing transferable features.
• The final prediction head concatenates the three embeddings (specific, generic, and metadata) and applies a fully connected layer with binary cross-entropy loss for multi-label classification.
• Data augmentation includes SpecAugment (frequency/time masking, warping), image-style augmentations (ShiftScaleRotate, Grid Distortion, Cutout), and Mixup with beta-distributed mixing.

Experimental results

Research questions

• RQ1 Can the integration of spatiotemporal metadata significantly improve performance in hierarchical urban sound tagging?
• RQ2 How do specific, task-learned audio embeddings compare to generic pre-trained embeddings in low-resource urban sound tagging scenarios?
• RQ3 Does replacing bi-GRU layers with a transformer encoder in a CRNN architecture lead to better performance and faster convergence?
• RQ4 To what extent does relabeling the training set using a high-performing model improve final system accuracy?
• RQ5 What is the optimal combination of loss functions (e.g., masked BCE for fine-grained tags) for hierarchical multilabel audio tagging?

Key findings

• System3 achieved a macro-AUPRC of 0.8107 on the coarse-grained tag classification, significantly outperforming the baseline score of 0.6323.
• On the fine-grained tag classification, System3 achieved a macro-AUPRC of 0.7040, surpassing the baseline of 0.5278.
• The use of both generic and specific audio embeddings, combined with metadata, led to consistent improvements over models using only one embedding type.
• The incorporation of Mixup and SpecAugment data augmentation techniques contributed to improved generalization and higher AUPRC scores.
• The transformer-based encoder in the specific embedding head reduced model parameters and improved performance compared to RNN-based alternatives.
• The relabeling strategy using a high-performing model (System2) improved the quality of training labels, contributing to better generalization on the validation set.