[Paper Review] Multi-Task Learning for Interpretable Weakly Labelled Sound Event Detection

Soham Deshmukh, Bhiksha Raj+1 more|arXiv (Cornell University)|Aug 17, 2020

Music and Audio Processing45 references4 citations

TL;DR

This paper proposes a multi-task learning framework for weakly labelled sound event detection (SED) that jointly optimizes SED with a time-frequency (T-F) reconstruction auxiliary task and a two-step attention pooling mechanism. The method improves SED performance by denoising internal T-F representations and enhancing interpretability, achieving 22.3%, 12.8%, and 5.9% relative F1-score improvements over a benchmark at 0, 10, and 20 dB SNR, respectively.

ABSTRACT

Weakly Labelled learning has garnered lot of attention in recent years due to its potential to scale Sound Event Detection (SED) and is formulated as Multiple Instance Learning (MIL) problem. This paper proposes a Multi-Task Learning (MTL) framework for learning from Weakly Labelled Audio data which encompasses the traditional MIL setup. To show the utility of proposed framework, we use the input TimeFrequency representation (T-F) reconstruction as the auxiliary task. We show that the chosen auxiliary task de-noises internal T-F representation and improves SED performance under noisy recordings. Our second contribution is introducing two step Attention Pooling mechanism. By having 2-steps in attention mechanism, the network retains better T-F level information without compromising SED performance. The visualisation of first step and second step attention weights helps in localising the audio-event in T-F domain. For evaluating the proposed framework, we remix the DCASE 2019 task 1 acoustic scene data with DCASE 2018 Task 2 sounds event data under 0, 10 and 20 db SNR resulting in a multi-class Weakly labelled SED problem. The proposed total framework outperforms existing benchmark models over all SNRs, specifically 22.3 %, 12.8 %, 5.9 % improvement over benchmark model on 0, 10 and 20 dB SNR respectively. We carry out ablation study to determine the contribution of each auxiliary task and 2-step Attention Pooling to the SED performance improvement. The code is publicly released

Motivation & Objective

• Address the scalability and annotation cost challenges of strongly labelled sound event detection by leveraging weakly labelled data.
• Improve the robustness and interpretability of weakly labelled SED models under noisy conditions.
• Develop a multi-task learning framework that enhances feature representation through an auxiliary reconstruction task.
• Introduce a two-step attention pooling mechanism to preserve fine-grained temporal and frequency information in T-F representations.
• Enable better localization and understanding of audio events through visualizable attention weights.

Proposed method

• Formulate weakly labelled SED as a multi-task learning problem with SED as the primary task and T-F log-mel spectrogram reconstruction as the auxiliary task.
• Use a shared encoder with 2D average pooling to extract hierarchical representations from input audio, followed by a two-step attention pooling mechanism.
• Apply first-step attention pooling to identify candidate time-frequency regions of interest across all audio events.
• Apply second-step attention pooling to refine and assign attention weights to specific events, producing final event-level predictions.
• Train the model end-to-end using binary cross-entropy loss for SED and mean squared error for reconstruction.
• Utilize a decoder head to reconstruct the input log-mel spectrogram from the shared encoder's representation, enforcing denoising of internal representations.

Experimental results

Research questions

• RQ1 Can a multi-task learning framework with T-F reconstruction as an auxiliary task improve robustness and performance in weakly labelled sound event detection under noisy conditions?
• RQ2 How does a two-step attention pooling mechanism enhance interpretability and localization accuracy in the time-frequency domain?
• RQ3 To what extent does the auxiliary reconstruction task contribute to denoising internal T-F representations and improving SED performance?
• RQ4 How does the proposed framework compare to existing benchmark models across varying signal-to-noise ratios (SNR)?
• RQ5 Can the denoised T-F representations from the auxiliary task be leveraged for downstream tasks like audio source separation?

Key findings

• The proposed framework achieves 22.3% relative improvement in F1-score over the benchmark model at 0 dB SNR.
• At 10 dB SNR, the model shows a 12.8% relative improvement in F1-score, demonstrating robustness to noise.
• At 20 dB SNR, the model still achieves a 5.9% relative improvement, indicating sustained performance gains under high noise.
• Ablation studies show that the two-step attention pooling mechanism is the primary driver of performance gain, contributing more than the auxiliary reconstruction task.
• The auxiliary reconstruction task contributes approximately 1% improvement in F1-score, indicating its role in denoising internal representations.
• Visualizations of attention weights confirm that the model localizes audio events in the time-frequency domain with high alignment to actual event onsets and durations.