[Paper Review] Guided multi-branch learning systems for sound event detection with sound separation

Yuxin Huang, Liwei Lin+6 more|arXiv (Cornell University)|Jul 21, 2020

Music and Audio Processing5 references6 citations

TL;DR

This paper proposes a guided multi-branch learning (MBL) system for sound event detection (SED) that integrates sound separation (SS) and improves performance via shared feature encoders with diverse pooling strategies and auxiliary branches. By combining attention pooling, global average/max pooling, and a sound event detection branch, the model achieves state-of-the-art results, with an ensemble F1 score of 0.497 on the public test set and 0.472 on the validation set when incorporating SS-SED systems.

ABSTRACT

In this paper, we describe in detail our systems for DCASE 2020 Task 4. The systems are based on the 1st-place system of DCASE 2019 Task 4, which adopts weakly-supervised framework with an attention-based embedding-level pooling module and a semi-supervised learning approach named guided learning. This year, we incorporate multi-branch learning (MBL) into the original system to further improve its performance. MBL uses different branches with different pooling strategies (including instance-level and embedding-level strategies) and different pooling modules (including attention pooling, global max pooling or global average pooling modules), which share the same feature encoder of the model. Therefore, multiple branches pursuing different purposes and focusing on different characteristics of the data can help the feature encoder model the feature space better and avoid over-fitting. To better exploit the strongly-labeled synthetic data, inspired by multi-task learning, we also employ a sound event detection branch. To combine sound separation (SS) with sound event detection (SED), we fuse the results of SED systems with SS-SED systems which are trained using separated sound output by an SS system. The experimental results prove that MBL can improve the model performance and using SS has great potential to improve the performance of SED ensemble system.

Motivation & Objective

• To improve sound event detection (SED) performance using weakly-labeled, unlabeled, and synthetic data.
• To integrate sound separation (SS) into SED systems to enhance detection accuracy.
• To mitigate overfitting and improve feature space modeling through multi-branch learning (MBL) with shared encoders.
• To leverage synthetic data more effectively by introducing a dedicated SED branch inspired by multi-task learning.
• To evaluate the impact of combining SS-SED systems with standard SED systems in ensemble models.

Proposed method

• The system uses a shared convolutional neural network (CNN) feature encoder across multiple branches with different pooling strategies: embedding-level (E-) and instance-level (I-) pooling.
• Each branch employs distinct pooling modules: attention pooling (ATP), global average pooling (GAP), or global max pooling (GMP), enabling diverse feature extraction.
• A guided learning framework is applied, where a professional teacher model (PT-model) and a promising student model (PS-model) jointly train using labeled and pseudo-labeled data.
• For unlabeled data, the PS-model uses pseudo-labels from the PT-model, and after a warm-up phase, the PS-model's predictions are used to refine the PT-model's pseudo-labels.
• A dedicated sound event detection branch is introduced to better exploit strongly-labeled synthetic data, following multi-task learning principles.
• Sound separation is integrated by training SED models on outputs from a baseline SS system, and results are fused in ensemble systems to improve overall performance.

Experimental results

Research questions

• RQ1 Can multi-branch learning with diverse pooling strategies improve sound event detection performance in weakly-supervised settings?
• RQ2 Does incorporating sound separation into SED systems lead to measurable performance gains when using ensemble learning?
• RQ3 Can auxiliary branches for SED improve the utilization of synthetic, strongly-labeled data in a weakly-supervised framework?
• RQ4 How does the combination of different pooling modules (e.g., ATP, GAP, GMP) affect model generalization and robustness?
• RQ5 Does using separated audio outputs from a sound separation system enhance the performance of SED ensemble systems?

Key findings

• The ensemble system incorporating both standard SED and SS-SED models achieved an F1 score of 0.495 on the public test set and 0.472 on the validation set, outperforming systems without SS.
• The best-performing ensemble system, SED-Ensemble (submitted), achieved an F1 score of 0.497 on the public test set and 0.467 on the validation set.
• Replacing three SS-SED models with real-data-trained models in the ensemble led to a drop in performance (F1: 0.485 on test set), indicating that SS-SED systems contribute unique, complementary features.
• Adding auxiliary branches (I-GMP, I-GAP) to the base E-ATP model improved performance, with the best individual model (E-ATP + I-GAP) achieving 0.451 F1 on validation set.
• The use of guided learning with pseudo-labeling from teacher and student models effectively leveraged unlabeled data, contributing to improved generalization.
• The results demonstrate that combining sound separation with SED through ensemble learning significantly enhances system robustness and accuracy.