[Paper Review] Large-Scale Weakly Labeled Semi-Supervised Sound Event Detection in Domestic Environments

Romain Serizel, Nicolas Turpault+2 more|arXiv (Cornell University)|Jul 27, 2018

Music and Audio ProcessingComputer Science10 references64 citations

TL;DR

This paper presents DCASE 2018 Task 4 on large-scale weakly labeled semi-supervised sound event detection (SED) in domestic environments, using a small set of weakly labeled clips (no time boundaries) and a large pool of unlabeled audio from YouTube. The method employs a two-stage CRNN-based approach: first training a clip-level classifier on weak labels, then fine-tuning a frame-level model using pseudo-labels from the first stage, achieving a macro F1-score of 14.06% on the test set, demonstrating the feasibility of semi-supervised SED with minimal human-annotated time boundaries.

ABSTRACT

This paper presents DCASE 2018 task 4. The task evaluates systems for the large-scale detection of sound events using weakly labeled data (without time boundaries). The target of the systems is to provide not only the event class but also the event time boundaries given that multiple events can be present in an audio recording. Another challenge of the task is to explore the possibility to exploit a large amount of unbalanced and unlabeled training data together with a small weakly labeled training set to improve system performance. The data are Youtube video excerpts from domestic context which have many applications such as ambient assisted living. The domain was chosen due to the scientific challenges (wide variety of sounds, time-localized events.. .) and potential industrial applications .

Motivation & Objective

• To develop sound event detection systems that can operate with minimal human-annotated time boundaries, reflecting real-world constraints where time-coded annotations are costly.
• To explore the effectiveness of semi-supervised learning by combining a small set of weakly labeled clips with large-scale unlabeled audio from YouTube.
• To improve time-boundary estimation for sound events, a key challenge in SED, by leveraging unlabeled data and frame-level modeling.
• To evaluate systems on a realistic, unbalanced dataset of domestic sound events with diverse durations and overlapping events.

Proposed method

• A two-stage deep learning pipeline is used: first, a CRNN is trained on weakly labeled clips (1,578 clips) to predict event presence at the clip level.
• Pseudo-labels are generated for 14,412 unlabeled in-domain clips using the first-stage model, which are then used to train a second CRNN for frame-level prediction.
• The second-stage model uses time-distributed dense layers to output event probabilities per frame, enabling precise onset and offset estimation.
• Median filtering (51 frames ≈ 1s) is applied to the frame-level outputs to smooth predictions and improve boundary localization.
• The system is trained with early stopping and uses a macro-averaged F1-score with 200ms collar tolerance on onsets and 20% of event length on offsets for evaluation.
• The evaluation uses the sed_eval toolbox with event-based metrics that heavily penalize boundary errors to emphasize localization accuracy.

Experimental results

Research questions

• RQ1 Can semi-supervised learning effectively improve sound event detection performance when only a small fraction of training data has time-boundary annotations?
• RQ2 How does leveraging large-scale unlabeled audio from YouTube impact the accuracy of time-boundary estimation in SED?
• RQ3 To what extent can a two-stage deep learning approach—first clip-level classification, then frame-level refinement—outperform weakly supervised baselines in boundary localization?
• RQ4 How do system performance and error patterns vary across sound events with different durations (e.g., short vs. long events)?
• RQ5 Does the inclusion of out-of-domain unlabeled data (e.g., speech-heavy clips) degrade or improve performance on the target domestic SED task?

Key findings

• The baseline system achieved a macro F1-score of 7.51% after the first pass on weakly labeled data, indicating limited performance in time-boundary detection.
• After the second pass, which fine-tuned the model on pseudo-labeled unlabeled data, the macro F1-score improved to 14.06%, demonstrating the benefit of semi-supervised learning.
• The largest performance gains were observed for longer events such as 'Vacuum cleaner' (46.5% F1) and 'Electric shaver/toothbrush' (32.4% F1), while short events like 'Cat' and 'Speech' remained poorly detected.
• The system struggled significantly with short events, achieving near-zero F1-scores for 'Cat', 'Dog', and 'Speech', highlighting the challenge of accurate boundary estimation.
• The metric's heavy penalty on boundary errors explains the low scores for short events, as even small onset/offset errors lead to both false positives and false negatives.
• The results confirm that time-segmentation remains a major bottleneck in SED, and that current smoothing techniques are insufficient for accurate localization.