[Paper Review] COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning

Simon Ging, Mohammadreza Zolfaghari+2 more|arXiv (Cornell University)|Nov 1, 2020

Multimodal Machine Learning ApplicationsComputer Science81 references75 citations

TL;DR

COOT introduces a hierarchical transformer with an attention-aware feature aggregation and a contextual transformer, plus a cross-modal cycle-consistency loss, to learn joint video-text embeddings with strong long-range temporal modeling and state-of-the-art retrieval/captioning performance.

ABSTRACT

Many real-world video-text tasks involve different levels of granularity, such as frames and words, clip and sentences or videos and paragraphs, each with distinct semantics. In this paper, we propose a Cooperative hierarchical Transformer (COOT) to leverage this hierarchy information and model the interactions between different levels of granularity and different modalities. The method consists of three major components: an attention-aware feature aggregation layer, which leverages the local temporal context (intra-level, e.g., within a clip), a contextual transformer to learn the interactions between low-level and high-level semantics (inter-level, e.g. clip-video, sentence-paragraph), and a cross-modal cycle-consistency loss to connect video and text. The resulting method compares favorably to the state of the art on several benchmarks while having few parameters. All code is available open-source at https://github.com/gingsi/coot-videotext

Motivation & Objective

• Motivate learning joint video-text embeddings that exploit multiple granularity levels (frame/word, clip/sentence, video/paragraph).
• Develop mechanisms to aggregate intra-level temporal context and model inter-level interactions between local and global semantics.
• Enforce cross-modal semantic alignment via a cycle-consistency loss to connect video and text representations.

Proposed method

• Introduce Attention-FA: an attention-aware feature aggregation layer that replaces simple pooling to capture intra-level temporal interactions.
• Propose Contextual Transformer (CoT) to model inter-level interactions between local (clip/sentence) and global contexts.
• Add a cross-modal cycle-consistency loss (CMC) to enforce semantic alignment between clips and sentences across modalities.
• Adopt hierarchical transformers with three levels: temporal transformer for frame/word features, attention-based aggregation for clip/sentence features, and contextual transformer for final embeddings.
• Use alignment losses at clip-sentence, video-paragraph, and global context levels following Zhang et al. (2021) and a cross-modal CMC loss (8) in the final objective.

Experimental results

Research questions

• RQ1 How can hierarchical modeling of videos and text improve cross-modal alignment and retrieval accuracy?
• RQ2 Do intra-level attention and inter-level contextual interactions improve representation quality for video-text tasks?
• RQ3 Does a cross-modal cycle-consistency loss improve semantic alignment between video and text embeddings?

Key findings

• COOT achieves state-of-the-art retrieval performance on ActivityNet-captions and YouCook2 datasets, with notable improvements over prior methods.
• An ablation study shows Attention-FA and Contextual Transformer (CoT), along with the CMC loss, each contribute to performance gains.
• On ActivityNet-captions, COOT demonstrates up to 16.6% average improvement in R@1 over the HSE baseline while using significantly fewer parameters (10.6M, about 60% fewer).
• On YouCook2, COOT outperforms several contemporary methods, especially when using features pretrained on HowTo100M, and benefits from hierarchy modeling beyond classification pretraining.
• COOT also demonstrates competitive video captioning results using learned representations, with fewer input features compared to other captioning pipelines.