[Paper Review] Sound Event Localization and Detection for Real Spatial Sound Scenes: Event-Independent Network and Data Augmentation Chains

Jinbo Hu, Yin Cao+5 more|arXiv (Cornell University)|Sep 5, 2022

Music and Audio Processing4 citations

TL;DR

This paper proposes an Event-Independent Network V2 (EINV2) with Conformer-based feature extraction and stochastic data augmentation chains for sound event localization and detection (SELD) in real spatial soundscapes. By leveraging simulated data from FSD50K, AudioSet, and TAU-SRIR DB, and applying multi-head self-attention with soft parameter sharing, the method achieves second place in the DCASE 2022 Task 3 challenge, demonstrating improved generalization and robustness to class imbalance and environmental variation.

ABSTRACT

Sound event localization and detection (SELD) is a joint task of sound event detection and direction-of-arrival estimation. In DCASE 2022 Task 3, types of data transform from computationally generated spatial recordings to recordings of real-sound scenes. Our system submitted to the DCASE 2022 Task 3 is based on our previous proposed Event-Independent Network V2 (EINV2) with a novel data augmentation method. Our method employs EINV2 with a track-wise output format, permutation-invariant training, and a soft parameter-sharing strategy, to detect different sound events of the same class but in different locations. The Conformer structure is used for extending EINV2 to learn local and global features. A data augmentation method, which contains several data augmentation chains composed of stochastic combinations of several different data augmentation operations, is utilized to generalize the model. To mitigate the lack of real-scene recordings in the development dataset and the presence of sound events being unbalanced, we exploit FSD50K, AudioSet, and TAU Spatial Room Impulse Response Database (TAU-SRIR DB) to generate simulated datasets for training. We present results on the validation set of Sony-TAu Realistic Spatial Soundscapes 2022 (STARSS22) in detail. Experimental results indicate that the ability to generalize to different environments and unbalanced performance among different classes are two main challenges. We evaluate our proposed method in Task 3 of the DCASE 2022 challenge and obtain the second rank in the teams ranking. Source code is released.

Motivation & Objective

• Address the challenge of sound event localization and detection (SELD) in real spatial soundscapes, where synthetic data distributions do not match real-world conditions.
• Overcome limitations in existing models, such as inability to distinguish same-class events at different locations (homogeneous overlap), by using an event-independent track-wise output format.
• Improve model generalization across diverse environments and mitigate performance degradation due to class imbalance in real-scene recordings.
• Develop a novel data augmentation strategy using stochastic chains of augmentation operations to increase feature diversity and robustness.
• Enhance model performance using external datasets (FSD50K, AudioSet) and measured room impulse responses (TAU-SRIR DB) to generate realistic simulated training data.

Proposed method

• Employ an Event-Independent Network V2 (EINV2) with a track-wise output format, where each track predicts a sound event independently, enabling detection of same-class events at different locations.
• Integrate multi-head self-attention (MHSA) and a soft parameter-sharing strategy across tracks to enable multi-task learning and improve generalization.
• Extend EINV2 with Conformer blocks—combining convolutional and self-attention mechanisms—to capture both local and global temporal-spectral features.
• Implement a novel data augmentation method using stochastic chains of augmentation operations, where multiple operations are randomly sampled and layered to generate diverse training samples.
• Generate simulated training data by stochastically convolving sound events from FSD50K and AudioSet with measured room impulse responses from the TAU-SRIR DB to mimic real spatial sound scenes.
• Use log-mel spectrograms and intensity vectors (IVs) in log-mel space as input features, computed directly from raw waveforms using 1D convolutional layers for end-to-end data augmentation support.

Experimental results

Research questions

• RQ1 How can a sound event localization and detection model be made robust to generalization across diverse real-world acoustic environments?
• RQ2 To what extent can data augmentation chains improve model generalization when training data is limited or mismatched to real test conditions?
• RQ3 Can an event-independent network with soft parameter sharing effectively detect multiple instances of the same sound class at different spatial locations?
• RQ4 How does the performance of a model trained on simulated data compare to one trained on official synthetic mixtures in real spatial soundscapes?
• RQ5 What are the primary causes of performance degradation in SELD systems when dealing with unbalanced class distributions and complex spatial relationships in real scenes?

Key findings

• The proposed system achieved the second-highest rank in the DCASE 2022 Task 3 challenge, outperforming the official baseline model.
• EINV2 with data augmentation chains significantly improved performance over EINV2 without augmentation, demonstrating the effectiveness of the stochastic augmentation chains.
• The model achieved an F1-score of 80.0% under a 20° spatial threshold for the 'knock' class, but only 2.2% for 'water tap and faucet', indicating strong class-wise performance imbalance.
• Localization performance was worst in room 2, likely due to its small size, and the 'walk, footsteps' class achieved 0.0% localization recall in room 24, suggesting issues with synthetic data quality.
• The system's performance was significantly degraded for the 'water tap and faucet' class in room 24, attributed to poor simulation of temporal and spatial co-occurrence with interfering events like 'dishes, pots, and pans'.
• Simulated data generated from FSD50K, AudioSet, and TAU-SRIR DB proved more effective than official synthetic mixtures, highlighting the value of realistic data augmentation.