[Paper Review] META-SELD: Meta-Learning for Fast Adaptation to the new environment in Sound Event Localization and Detection

Jinbo Hu, Yin Cao+6 more|arXiv (Cornell University)|Aug 17, 2023

Speech and Audio Processing4 citations

TL;DR

This paper proposes Meta-SELD, a meta-learning framework based on Model-Agnostic Meta-Learning (MAML) to enable fast adaptation of sound event localization and detection (SELD) models to unseen acoustic environments. By learning general initial parameters across diverse environments, Meta-SELD achieves significant performance gains—especially in localization accuracy—using only a few samples and minimal fine-tuning steps, outperforming standard fine-tuning on the STARSS23 dataset.

ABSTRACT

For learning-based sound event localization and detection (SELD) methods, different acoustic environments in the training and test sets may result in large performance differences in the validation and evaluation stages. Different environments, such as different sizes of rooms, different reverberation times, and different background noise, may be reasons for a learning-based system to fail. On the other hand, acquiring annotated spatial sound event samples, which include onset and offset time stamps, class types of sound events, and direction-of-arrival (DOA) of sound sources is very expensive. In addition, deploying a SELD system in a new environment often poses challenges due to time-consuming training and fine-tuning processes. To address these issues, we propose Meta-SELD, which applies meta-learning methods to achieve fast adaptation to new environments. More specifically, based on Model Agnostic Meta-Learning (MAML), the proposed Meta-SELD aims to find good meta-initialized parameters to adapt to new environments with only a small number of samples and parameter updating iterations. We can then quickly adapt the meta-trained SELD model to unseen environments. Our experiments compare fine-tuning methods from pre-trained SELD models with our Meta-SELD on the Sony-TAU Realistic Spatial Soundscapes 2023 (STARSSS23) dataset. The evaluation results demonstrate the effectiveness of Meta-SELD when adapting to new environments.

Motivation & Objective

• To address the challenge of performance degradation in SELD models when deployed in unseen acoustic environments such as different rooms, reverberation times, or noise levels.
• To reduce the high cost and effort of collecting and annotating spatial sound event data by enabling few-shot adaptation to new environments.
• To accelerate deployment of SELD systems in new environments by minimizing the need for extensive retraining or fine-tuning.
• To improve generalization and robustness of SELD models across diverse acoustic conditions using meta-learning.
• To achieve faster and more effective adaptation than standard fine-tuning using only a small number of environment-specific samples.

Proposed method

• Meta-SELD employs Model-Agnostic Meta-Learning (MAML) to learn a shared, general-purpose initialization of model parameters across multiple acoustic environments.
• Each unique recording environment (e.g., specific room with its room impulse response) is treated as a distinct task in the meta-learning process.
• The model is trained to quickly adapt to a new environment using only a few support samples and a small number of gradient update steps.
• The method uses synthetic data: 1-minute audio clips convolved with multi-channel spatial room impulse responses (SRIRs) from the TAU-SRIR DB, using sound events from FSD50K and AudioSet filtered via PANNs.
• The SELD model uses the Activity-coupled Cartesian DOA (ACCDOA) representation, which combines sound event detection and direction-of-arrival estimation into a single output branch.
• Meta-SELD optimizes for a meta-objective that minimizes the average loss across all tasks (environments), enabling fast adaptation to unseen environments during inference.

Figure 1: The multi-ACCDOA representation of the SELD model. There is no track dimension in the ACCDOA representation.

Experimental results

Research questions

• RQ1 Can meta-learning reduce the adaptation time and performance drop of SELD models in unseen acoustic environments?
• RQ2 How does Meta-SELD compare to standard fine-tuning when only a few samples from a new environment are available?
• RQ3 Does meta-learning improve localization accuracy, especially in environments where baseline models perform poorly?
• RQ4 To what extent does Meta-SELD generalize across diverse acoustic conditions such as different room sizes, reverberation, and noise levels?
• RQ5 Can Meta-SELD reduce overfitting risks when fine-tuning on small, unseen environment datasets?

Key findings

• Meta-SELD significantly improves the overall SELD metric (ℰ_SELD) across all rooms in the STARSS23 dataset compared to both pre-trained models and fine-tuning baselines.
• In fold4_room2—where the pre-trained model and fine-tuning method showed poor localization performance—Meta-SELD reduced the class-dependent localization error (LE_CD) by 14.8° compared to the pre-trained model and by 8.3° compared to fine-tuning.
• For fold3_room22 and fold4_room23, Meta-SELD achieved the highest scores in ER≤20°, F≤20°, and LR_CD, outperforming other methods, particularly in sound event detection.
• Meta-SELD improved F≤20° and reduced ER≤20° in fold4_room2 due to better DOA estimation, directly resulting from reduced LE_CD.
• Despite improvements in several rooms, Meta-SELD showed performance degradation in fold3_room4, fold3_room7, fold3_room14, and fold4_room16, primarily due to declining SED performance and potential optimization conflicts across tasks.
• The method demonstrates robustness in low-data regimes, reducing overfitting risks compared to standard fine-tuning when using only a few samples from a new environment.