[Paper Review] AVA-Speech: A Densely Labeled Dataset of Speech Activity in Movies

Sourish Chaudhuri, Joseph Roth+9 more|arXiv (Cornell University)|Aug 2, 2018

Speech and Audio Processing28 references4 citations

TL;DR

This paper introduces AVA-Speech, a publicly released, densely labeled dataset of speech activity in 190 movies totaling ~46 hours, with frame-level annotations for clean speech, speech with music, and speech with noise. It evaluates state-of-the-art audio and visual models, showing that audio models degrade under noise while visual speech classification (VSC) remains stable, establishing a benchmark for future research in audio-visual speech detection.

ABSTRACT

Speech activity detection (or endpointing) is an important processing step for applications such as speech recognition, language identification and speaker diarization. Both audio- and vision-based approaches have been used for this task in various settings, often tailored toward end applications. However, much of the prior work reports results in synthetic settings, on task-specific datasets, or on datasets that are not openly available. This makes it difficult to compare approaches and understand their strengths and weaknesses. In this paper, we describe a new dataset which we will release publicly containing densely labeled speech activity in YouTube videos, with the goal of creating a shared, available dataset for this task. The labels in the dataset annotate three different speech activity conditions: clean speech, speech co-occurring with music, and speech co-occurring with noise, which enable analysis of model performance in more challenging conditions based on the presence of overlapping noise. We report benchmark performance numbers on AVA-Speech using off-the-shelf, state-of-the-art audio and vision models that serve as a baseline to facilitate future research.

Motivation & Objective

• To address the lack of a standard, publicly available benchmark dataset for speech activity detection (VAD) in real-world, diverse media contexts.
• To create a large-scale, temporally dense dataset with video and audio annotations to support evaluation of both audio-only and audio-visual VAD models.
• To explicitly label speech activity under three challenging conditions: clean speech, speech with music, and speech with noise, enabling analysis of model robustness.
• To provide baseline performance results using off-the-shelf state-of-the-art audio and visual models for future comparison.
• To establish a shared benchmark that supports future research in joint audio-visual modeling and extension to related tasks.

Proposed method

• The dataset is constructed from 185 movie clips (15 minutes each) sourced from YouTube, totaling ~46 hours of video, selected to represent diverse speaking contexts.
• Human raters performed dense, frame-level labeling of speech activity, categorizing each frame as: NoSpeech, CleanSpeech, SpeechWithMusic, or SpeechWithNoise, based on predefined instructions.
• Audio-based models were evaluated using WebRTC VAD, a tiny CNN (tiny_320), and a larger ResNet-50-based model (resnet_960) on 10ms mel-spectrogram inputs.
• Visual speech classification (VSC) was performed by detecting and tracking faces, then applying a lightweight CNN on 3-frame sequences of face thumbnails to predict speech activity.
• Model performance was evaluated at a fixed false positive rate (FPR = 0.315) across all conditions, with TPR reported per condition and overall.
• ROC curves were generated by varying classification thresholds to assess performance across operating points, with latency measured on a single-threaded CPU.

Experimental results

Research questions

• RQ1 How do state-of-the-art audio and visual speech detection models perform on a real-world, diverse movie dataset with varying background noise conditions?
• RQ2 To what extent does background noise (e.g., music or ambient noise) degrade the performance of audio-only speech activity detection models?
• RQ3 Can visual speech classification models maintain consistent performance across different noise conditions where audio models fail?
• RQ4 How does the performance of audio and visual models compare when evaluated under a fixed false positive rate, reflecting real-world deployment constraints?
• RQ5 What is the inference latency of these models, and how does it impact practical deployment in real-time systems?

Key findings

• The WebRTC VAD achieved a true positive rate (TPR) of 0.786 on clean speech, but dropped to 0.706 on speech with noise and 0.733 overall, indicating sensitivity to background noise.
• The tiny_320 audio model achieved a TPR of 0.965 on clean speech, 0.826 on speech with music, and 0.623 on speech with noise, showing significant degradation under noise.
• The larger resnet_960 model achieved the highest TPR of 0.992 on clean speech and 0.944 on speech with music, but only 0.787 on speech with noise, confirming audio model degradation under noise.
• The visual speech classification (VSC) model achieved a TPR of 0.588 on clean speech, 0.568 on speech with music, and 0.556 on speech with noise, remaining relatively stable across conditions.
• Despite lower absolute performance, VSC showed consistent performance across noise types, suggesting robustness in challenging acoustic environments.
• The VSC model had a latency of 40.0 ms, significantly higher than audio models (0.06–2.23 ms), primarily due to face detection and tracking overhead.