[Paper Review] How2: A Large-scale Dataset for Multimodal Language Understanding

Ramon Sanabria, Ozan Çağlayan+5 more|arXiv (Cornell University)|Nov 1, 2018

Natural Language Processing TechniquesComputer Science35 references152 citations

TL;DR

How2 introduces a large-scale, multilingual multimodal dataset of instructional videos with English subtitles, word-level alignments, and Portuguese translations, plus baselines showing multimodal benefits for ASR, MT, STT, and summarization.

ABSTRACT

In this paper, we introduce How2, a multimodal collection of instructional videos with English subtitles and crowdsourced Portuguese translations. We also present integrated sequence-to-sequence baselines for machine translation, automatic speech recognition, spoken language translation, and multimodal summarization. By making available data and code for several multimodal natural language tasks, we hope to stimulate more research on these and similar challenges, to obtain a deeper understanding of multimodality in language processing.

Motivation & Objective

• Provide a large-scale, naturally occurring multimodal dataset linking speech, text, and vision across instructional topics.
• Enable multilingual (English-Portuguese) annotations to support cross-lingual multimodal tasks.
• Offer baseline models and tasks to study the impact of visual context on ASR, MT, STT, and summarization.

Proposed method

• Assemble a dataset of 79,114 instructional videos (~2,000 hours) with English subtitles and ground-truth English summaries.
• Crowdsource Portuguese translations of subtitles via post-editing of machine-translated segments, focusing on quality controls.
• Extract per-clip visual features and align word-level subtitles to audio using Viterbi alignment with a Kaldi WSJ model.
• Train sequence-to-sequence baselines for ASR, MT English→Portuguese, STT, and multimodal summarization using hierarchical attention.
• Apply visual-adaptive training by learning a video-specific bias for speech features in multimodal tasks.
• Evaluate with WER for ASR, BLEU for MT/STT, and ROUGE-L for summarization.

Experimental results

Research questions

• RQ1 Can multimodal information (especially action-level visual cues) improve performance on ASR, MT, STT, and summarization tasks compared to text-only baselines?
• RQ2 How does visualization influence cross-language understanding in English↔Portuguese multimodal settings?
• RQ3 What are the data characteristics and alignment quality when pairing speech, text, and video in a large-scale instructional-domain corpus?

Key findings

• Multimodal models achieve better ASR performance (WER reduced from 19.4 to 18.0).
• Multimodal MT shows no change in BLEU compared to baseline (54.4) in this setup, but multimodal MT with adaptive inputs maintains parity.
• Multimodal STT improves BLEU from 36.0 to 37.2.
• Multimodal summarization improves ROUGE-L from 53.9 to 54.9.
• The 300h subset contains about 3.8M English words and 3.6M Portuguese words; the full 2000h corpus has ~22.5M English words.
• Visual features (action-level) and hierarchical attention contribute to gains in multimodal tasks.