[Paper Review] HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training

Linjie Li, Yen-Chun Chen+4 more|arXiv (Cornell University)|May 1, 2020

Multimodal Machine Learning Applications76 references51 citations

TL;DR

HERO introduces a hierarchical video+language pre-training model with a Cross-modal Transformer for local fusion and a Temporal Transformer for global context, plus new pre-training tasks (VSM and FOM) to learn temporal alignment; it achieves state-of-the-art on multiple video+language tasks and introduces new benchmarks How2R and How2QA.

ABSTRACT

We present HERO, a novel framework for large-scale video+language omni-representation learning. HERO encodes multimodal inputs in a hierarchical structure, where local context of a video frame is captured by a Cross-modal Transformer via multimodal fusion, and global video context is captured by a Temporal Transformer. In addition to standard Masked Language Modeling (MLM) and Masked Frame Modeling (MFM) objectives, we design two new pre-training tasks: (i) Video-Subtitle Matching (VSM), where the model predicts both global and local temporal alignment; and (ii) Frame Order Modeling (FOM), where the model predicts the right order of shuffled video frames. HERO is jointly trained on HowTo100M and large-scale TV datasets to gain deep understanding of complex social dynamics with multi-character interactions. Comprehensive experiments demonstrate that HERO achieves new state of the art on multiple benchmarks over Text-based Video/Video-moment Retrieval, Video Question Answering (QA), Video-and-language Inference and Video Captioning tasks across different domains. We also introduce two new challenging benchmarks How2QA and How2R for Video QA and Retrieval, collected from diverse video content over multimodalities.

Motivation & Objective

• Develop a scalable, multimodal pre-training framework that captures fine-grained frame-text alignment and global video context.
• Leverage a hierarchical encoder to fuse subtitles with local frame context and extract global temporal representations.
• Design pre-training tasks that enforce both local and global temporal alignment between video and language.
• Diversify pre-training data with TV shows and HowTo100M to learn complex social dynamics and narratives.
• Introduce new benchmarks for video QA and retrieval to evaluate multimodal understanding in diverse content.

Proposed method

• Propose a hierarchical model with a Cross-modal Transformer for local fusion of subtitle sentences and their associated video frames.
• Use a Temporal Transformer to compute sequential, global video context from cross-modal outputs.
• Introduce four pre-training tasks: Masked Language Modeling (MLM), Masked Frame Modeling (MFM) with two variants (MFFR and MNCE), Video-Subtitle Matching (VSM), and Frame Order Modeling (FOM).
• In MLM, predict masked tokens using surrounding text and aligned visual frames; in MFM, reconstruct or distinguish masked video features using NCE-based objectives; in VSM, learn local and global subtitle-video alignment; in FOM, predict original frame order after shuffling a subset of frames.
• Pre-train on TV and HowTo100M datasets, with ablations and comparisons to state-of-the-art models across multiple downstream tasks.

Experimental results

Research questions

• RQ1 Can a hierarchical video+language encoder better exploit temporal alignment between subtitles and video frames than flat architectures?
• RQ2 Do new pre-training tasks (VSM and FOM) improve temporal reasoning and alignment for downstream tasks like video question answering and retrieval?
• RQ3 How does training on diverse datasets (TV shows and instructional videos) affect performance on varied video+language benchmarks?
• RQ4 What is the impact of the proposed pre-training configuration (MLM + MNCE + FOM + VSM) on downstream tasks compared to baselines?
• RQ5 Do the proposed benchmarks How2R and How2QA effectively evaluate rich multimodal understanding in video content?

Key findings

• Hero achieves new state-of-the-art results on multiple downstream tasks including text-based video retrieval, video QA, and video-language inference.
• The optimal pre-training setup MLM + MNCE + FOM + VSM yields strong performance across TVR, TVQA, How2R, and How2QA.
• The hierarchical encoder with explicit local (Cross-modal) and global (Temporal) fusion outperforms flat BERT-like baselines.
• VSM improves video-moment retrieval by learning both local and global subtitle-video alignment, especially benefiting retrieval tasks.
• FOM helps temporal reasoning in QA tasks, and MFFR provides limited additional gains when MNCE is used.
• Hero generalizes well to both multi-channel (video+subtitle) and single-channel (video-only) settings, outperforming task-specific SOTA models.