[Paper Review] Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research

Atousa Torabi, Christopher Pal+2 more|arXiv (Cornell University)|Mar 3, 2015

Multimodal Machine Learning ApplicationsComputer Science13 references160 citations

TL;DR

This paper presents a large-scale, DVS-derived video annotation dataset (M-VAD) created by automatically segmenting and aligning Descriptive Video Service audio tracks from 92 DVDs with minimal human intervention. Using audio-based DVS narration isolation and pause detection for temporal alignment, the authors produce a high-quality dataset of 84.6 hours of video paired with professionally written, temporally aligned natural language descriptions, which enables training of deep learning models for video description with improved semantic and syntactic richness.

ABSTRACT

In this work, we introduce a dataset of video annotated with high quality natural language phrases describing the visual content in a given segment of time. Our dataset is based on the Descriptive Video Service (DVS) that is now encoded on many digital media products such as DVDs. DVS is an audio narration describing the visual elements and actions in a movie for the visually impaired. It is temporally aligned with the movie and mixed with the original movie soundtrack. We describe an automatic DVS segmentation and alignment method for movies, that enables us to scale up the collection of a DVS-derived dataset with minimal human intervention. Using this method, we have collected the largest DVS-derived dataset for video description of which we are aware. Our dataset currently includes over 84.6 hours of paired video/sentences from 92 DVDs and is growing.

Motivation & Objective

• To address the lack of large-scale, high-quality video description datasets for training deep learning models.
• To develop an automated method for extracting and aligning DVS narrations from DVDs with minimal human intervention.
• To create a publicly available, balanced dataset for video description research that supports diverse genres and professional-level descriptions.
• To improve the quality and alignment of video-text pairs compared to existing methods relying on movie scripts or crowdsourced annotations.
• To enable training of end-to-end video description models using professionally written, temporally precise descriptions from DVS audio tracks.

Proposed method

• Automatically isolate DVS narration segments from mixed audio tracks using audio signal processing and pause detection to identify natural segmentation points.
• Use audio similarity analysis via Fast Fourier Transform (FFT) to detect DVS segments and improve alignment with video content.
• Apply a two-second temporal buffer to each video clip to compensate for potential 1–2 second misalignment between DVS narration and visual content.
• Transcribe DVS audio with over 98% accuracy using a hybrid automatic speech recognition and human transcription pipeline.
• Replace all proper names in the text with a single token (e.g., 'SOMEONE') to reduce model overfitting and improve generalization.
• Create a balanced train/validation/test split (38,949 / 4,888 / 5,149 clips) that preserves genre distribution across sets.

Experimental results

Research questions

• RQ1 Can DVS audio tracks from commercial DVDs be automatically segmented and aligned with video content to create a large-scale, high-quality video description dataset?
• RQ2 How does the quality and alignment of DVS-derived descriptions compare to those derived from movie scripts or crowdsourced annotations?
• RQ3 To what extent does removing proper names from the dataset improve the performance of video description models?
• RQ4 Can an LSTM-based video description model trained on this dataset generate semantically meaningful and contextually relevant descriptions?
• RQ5 Does the use of professionally written, naturally paced DVS narrations lead to better generalization in video description tasks compared to synthetic or crowd-sourced captions?

Key findings

• The M-VAD dataset contains 84.6 hours of video paired with 55,904 natural language descriptions from 92 DVDs, with a median clip duration of 6.2 seconds.
• The dataset is the largest DVS-derived video description dataset known, surpassing prior work that used only 46 DVDs and combined scripts with DVS.
• The DVS-derived descriptions show high temporal alignment, with a maximum misalignment of 2 seconds, and are written in rich, syntactically complex sentences.
• The vocabulary contains 17,609 unique words, with 9,512 nouns, 2,571 verbs, 3,560 adjectives, and 857 adverbs, indicating detailed visual description of scene elements.
• The 10 most frequent verbs include five synonyms of 'seeing' (e.g., gaze, look, stare), reflecting the focus on visual attention and observation in the narration.
• Qualitative results show that an LSTM model trained on the dataset generates meaningful, contextually relevant descriptions that closely match the original DVS narrations.