[Paper Review] Weakly supervised CRNN system for sound event detection with large-scale unlabeled in-domain data

Dezhi Wang, Lilun Zhang+4 more|arXiv (Cornell University)|Nov 1, 2018

Music and Audio Processing6 citations

TL;DR

This paper proposes a weakly supervised CRNN system that leverages large-scale unlabeled in-domain audio data to improve sound event detection (SED). By using a pre-trained audio tagging model to generate pseudo-labels for unlabeled data and applying an ensemble strategy to mitigate noisy labels, the method achieves a 21.0% F1-score on the DCASE 2018 SED challenge, representing a 10% improvement over the baseline.

ABSTRACT

Sound event detection (SED) is typically posed as a supervised learning problem requiring training data with strong temporal labels of sound events. However, the production of datasets with strong labels normally requires unaffordable labor cost. It limits the practical application of supervised SED methods. The recent advances in SED approaches focuses on detecting sound events by taking advantages of weakly labeled or unlabeled training data. In this paper, we propose a joint framework to solve the SED task using large-scale unlabeled in-domain data. In particular, a state-of-the-art general audio tagging model is first employed to predict weak labels for unlabeled data. On the other hand, a weakly supervised architecture based on the convolutional recurrent neural network (CRNN) is developed to solve the strong annotations of sound events with the aid of the unlabeled data with predicted labels. It is found that the SED performance generally increases as more unlabeled data is added into the training. To address the noisy label problem of unlabeled data, an ensemble strategy is applied to increase the system robustness. The proposed system is evaluated on the SED dataset of DCASE 2018 challenge. It reaches a F1-score of 21.0%, resulting in an improvement of 10% over the baseline system.

Motivation & Objective

• To address the high annotation cost of strong temporal labels in sound event detection (SED) by leveraging large-scale unlabeled in-domain audio data.
• To develop a joint learning framework that combines weakly supervised CRNN with pseudo-labeled unlabeled data to improve SED performance.
• To mitigate the impact of noisy pseudo-labels from weakly labeled data through an ensemble strategy for enhanced robustness.
• To demonstrate the effectiveness of unsupervised pre-training and self-training with in-domain data on real-world SED benchmarks.

Proposed method

• A state-of-the-art audio tagging model is used to predict weak labels for large-scale unlabeled in-domain audio data.
• A weakly supervised CRNN architecture is trained using the predicted pseudo-labels to localize sound events with strong temporal annotations.
• An ensemble strategy is applied across multiple models to reduce the impact of noisy pseudo-labels and improve generalization.
• The system jointly optimizes the CRNN for SED using both the weakly labeled data and the original labeled training set.
• The framework is trained end-to-end with a joint loss combining CRNN outputs and pseudo-label confidence.

Experimental results

Research questions

• RQ1 Can large-scale unlabeled in-domain audio data significantly improve SED performance when combined with weak supervision?
• RQ2 How effective is pseudo-labeling from a pre-trained audio tagging model in reducing the need for costly strong annotations?
• RQ3 To what extent does an ensemble strategy mitigate the noise introduced by pseudo-labels in weakly supervised SED?
• RQ4 Does joint training with labeled and pseudo-labeled data lead to better generalization than supervised training alone?

Key findings

• The proposed system achieves a F1-score of 21.0% on the DCASE 2018 sound event detection evaluation set.
• The performance improvement over the baseline system is 10 percentage points, demonstrating the effectiveness of the proposed method.
• System performance consistently improves as more unlabeled data is incorporated into the training process.
• The ensemble strategy significantly enhances robustness, particularly in handling noisy pseudo-labels from the audio tagging model.