[Paper Review] AutoAD: Movie Description in Context

Tengda Han, Max Bain+4 more|arXiv (Cornell University)|Mar 29, 2023

Subtitles and Audiovisual MediaArts and Humanities3 citations

TL;DR

This paper proposes AutoAD, a novel framework for automatic movie audio description generation that leverages visual and contextual cues—such as prior AD, subtitles, and temporal context—by bridging pretrained vision-language models (CLIP) and large language models (GPT). The method achieves state-of-the-art performance on movie AD benchmarks through context-aware generation, data pretraining on partial-data sources, and a cleaned, enhanced MAD-v2 dataset, with strong zero-shot results on LSMDC.

ABSTRACT

The objective of this paper is an automatic Audio Description (AD) model that ingests movies and outputs AD in text form. Generating high-quality movie AD is challenging due to the dependency of the descriptions on context, and the limited amount of training data available. In this work, we leverage the power of pretrained foundation models, such as GPT and CLIP, and only train a mapping network that bridges the two models for visually-conditioned text generation. In order to obtain high-quality AD, we make the following four contributions: (i) we incorporate context from the movie clip, AD from previous clips, as well as the subtitles; (ii) we address the lack of training data by pretraining on large-scale datasets, where visual or contextual information is unavailable, e.g. text-only AD without movies or visual captioning datasets without context; (iii) we improve on the currently available AD datasets, by removing label noise in the MAD dataset, and adding character naming information; and (iv) we obtain strong results on the movie AD task compared with previous methods.

Motivation & Objective

• Address the challenge of generating high-quality, contextually coherent audio descriptions for movies, which require understanding of visual content, character identity, and temporal narrative flow.
• Overcome the scarcity of large-scale, high-quality movie AD training data by developing a data curation pipeline and leveraging pretraining on partially labeled or text-only data.
• Improve model performance by incorporating multi-source context: visual frames, prior AD outputs, and subtitles, especially for character naming and action description.
• Enable zero-shot generalization to downstream benchmarks like LSMDC by adapting the model to the style and structure of movie AD through targeted pretraining.
• Enhance the quality and reliability of existing MAD dataset annotations by removing label noise and adding character naming information, resulting in a cleaner, more usable benchmark.

Proposed method

• Integrate CLIP (vision encoder) and GPT-2 (language decoder) via a lightweight adapter network to enable visually conditioned text generation with contextual awareness.
• Incorporate multiple context types: prior AD sentences (recurrent context), subtitles (dialogue context), and visual frames to improve narrative coherence and entity tracking.
• Pretrain the vision encoder on open-domain visual-text datasets (e.g., WebVid, CC3M) and the language model on text-only AD data (AudioVault-AD) to bridge domain gaps and improve style alignment.
• Use a speaker-based separation pipeline to automatically collect large-scale text-only movie AD data for pretraining, enabling scalable data collection without full video access.
• Apply a recurrent context modeling strategy that processes AD outputs from previous clips as input, simulating real-time generation with limited context visibility.
• Optimize the model with a hybrid training setup: fine-tune on MAD-v2 with full context, and evaluate under both oracle (ground-truth context) and recurrent (predicted context) settings.

Experimental results

Research questions

• RQ1 Can a vision-language foundation model combined with context-aware adaptation improve the quality and coherence of automatic movie audio descriptions?
• RQ2 To what extent does pretraining on partially labeled or text-only data improve generalization in low-resource movie AD generation?
• RQ3 How does incorporating multiple context types—prior AD, subtitles, and visual frames—affect model performance and narrative consistency?
• RQ4 Can a model trained on a cleaned, enhanced version of the MAD dataset (MAD-v2) outperform previous methods despite fewer training samples?
• RQ5 How well does the proposed method generalize to zero-shot evaluation on the LSMDC benchmark without fine-tuning on its specific training data?

Key findings

• The AutoAD model achieves a CIDEr score of 21.9 on the MAD-v2 dataset under the oracle setting, significantly outperforming prior methods like ClipCap and CapDec.
• Pretraining on text-only AD data (AudioVault-AD) improves CIDEr from 12.6 to 14.1, demonstrating the importance of domain-specific language adaptation.
• The combination of visual-only pretraining (WebVid) and text-only pretraining (AudioVault-AD) yields a 21.9 CIDEr score, a 2.9-point gain over the baseline without pretraining.
• Using MAD-v2 instead of MAD-v1 improves performance across all settings, with a 1.5–2.0 CIDEr point gain despite fewer training samples, confirming reduced label noise and improved data quality.
• The model achieves a 16.7 CIDEr score on the LSMDC benchmark in a zero-shot setting, outperforming the previous SOTA method (15.4) without any fine-tuning on the LSMDC dataset.
• Increasing context length from 1 to 6 AD sentences improves performance, though gains plateau beyond 3 sentences due to attention complexity in GPT-2, indicating a trade-off between context and efficiency.