[Paper Review] VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research

Xin Wang, Jiawei Wu+4 more|arXiv (Cornell University)|Apr 6, 2019

Multimodal Machine Learning ApplicationsComputer Science74 references39 citations

TL;DR

VaTeX introduces a large-scale multilingual video description dataset (English and Chinese) with parallel translations, and proposes multilingual video captioning and video-guided machine translation tasks, showing multilingual models are effective and that video context aids translation.

ABSTRACT

We present a new large-scale multilingual video description dataset, VATEX, which contains over 41,250 videos and 825,000 captions in both English and Chinese. Among the captions, there are over 206,000 English-Chinese parallel translation pairs. Compared to the widely-used MSR-VTT dataset, VATEX is multilingual, larger, linguistically complex, and more diverse in terms of both video and natural language descriptions. We also introduce two tasks for video-and-language research based on VATEX: (1) Multilingual Video Captioning, aimed at describing a video in various languages with a compact unified captioning model, and (2) Video-guided Machine Translation, to translate a source language description into the target language using the video information as additional spatiotemporal context. Extensive experiments on the VATEX dataset show that, first, the unified multilingual model can not only produce both English and Chinese descriptions for a video more efficiently, but also offer improved performance over the monolingual models. Furthermore, we demonstrate that the spatiotemporal video context can be effectively utilized to align source and target languages and thus assist machine translation. In the end, we discuss the potentials of using VATEX for other video-and-language research.

Motivation & Objective

• Provide a large, multilingual benchmark for video captioning and translation.
• Enable multilingual research by pairing English and Chinese captions with a diverse set of videos.
• Investigate whether a compact multilingual model can describe videos in multiple languages more efficiently and effectively.
• Explore the potential of using video context to improve machine translation across languages.

Proposed method

• Construct VaTeX with 41,269 valid video clips, 825k captions (English and Chinese), including 206k English-Chinese parallel pairs.
• Annotate 10 English and 10 Chinese captions per video, with translations obtained via post-editing to ensure alignment with video content.
• Propose three multilingual captioning model variants: two monolingual baselines, Shared Enc (shared video encoder, language decoders), and Shared Enc-Dec (shared encoder and decoder with language-specific word embeddings).
• Adopt an attention-based encoder–decoder framework for captioning using 3D ConvNet (I3D) features and bidirectional LSTM encoders with dot-product attention.
• Introduce Video-Guided Machine Translation (VMT) using a multimodal seq2seq model with text and spatiotemporal video context and dual attention (source text and video).
• Evaluate models with BLEU-4, METEOR, ROUGE-L, and CIDEr on VaTeX-English and VaTeX-Chinese.

Experimental results

Research questions

• RQ1 Does multilingual training improve video captioning quality compared to monolingual models?
• RQ2 Can a compact unified multilingual model describe videos in English and Chinese effectively with shared encoders/embedders?
• RQ3 Does incorporating spatiotemporal video context improve machine translation between English and Chinese (Video-Guided Machine Translation)?
• RQ4 How does video context help recover nouns and verbs when translating across languages?
• RQ5 What are the lexical and diversity characteristics of VaTeX captions compared with existing datasets like MSR-VTT?

Key findings

• VaTeX is larger and multilingual, with 41.3k videos and 825k captions (English and Chinese), including 206k English–Chinese translation pairs.
• Multilingual models (Shared Enc and Shared Enc-Dec) consistently outperform monolingual baselines in BLEU-4, METEOR, ROUGE-L, and CIDEr, while dramatically reducing parameters (Shared Enc −4.7M and Shared Enc-Dec −13.4M).
• Video context improves translation performance: VMT (LSTM video features with temporal attention) yields BLEU-4 gains of +2.27 (English→Chinese) and +2.11 (Chinese→English) over strong baselines.
• Noun/verb masking experiments show VMT better recovers missing lexical items than NMT as masking increases, demonstrating video information helps disambiguation and alignment.
• VaTeX captions are longer and more lexically diverse than MSR-VTT, with fewer duplicates (no intra-video duplicates) and richer noun/verb usage across English and Chinese.