[Paper Review] HERO: HiErarchical spatio-tempoRal reasOning with Contrastive Action Correspondence for End-to-End Video Object Grounding

Mengze Li, Tianbao Wang+7 more|arXiv (Cornell University)|Aug 11, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

HERO proposes a novel end-to-end framework for video object grounding that addresses incomplete cross-modal correspondence and insensitive spatio-temporal locality by introducing weakly-supervised contrastive learning for action-consistent frame retrieval and hierarchical multi-head attention with pyramid and shifted alignment mechanisms. It achieves state-of-the-art performance on two benchmark datasets, outperforming prior methods by significant margins.

ABSTRACT

Video Object Grounding (VOG) is the problem of associating spatial object regions in the video to a descriptive natural language query. This is a challenging vision-language task that necessitates constructing the correct cross-modal correspondence and modeling the appropriate spatio-temporal context of the query video and caption, thereby localizing the specific objects accurately. In this paper, we tackle this task by a novel framework called HiErarchical spatio-tempoRal reasOning (HERO) with contrastive action correspondence. We study the VOG task at two aspects that prior works overlooked: (1) Contrastive Action Correspondence-aware Retrieval. Notice that the fine-grained video semantics (e.g., multiple actions) is not totally aligned with the annotated language query (e.g., single action), we first introduce the weakly-supervised contrastive learning that classifies the video as action-consistent and action-independent frames relying on the video-caption action semantic correspondence. Such a design can build the fine-grained cross-modal correspondence for more accurate subsequent VOG. (2) Hierarchical Spatio-temporal Modeling Improvement. While transformer-based VOG models present their potential in sequential modality (i.e., video and caption) modeling, existing evidence also indicates that the transformer suffers from the issue of the insensitive spatio-temporal locality. Motivated by that, we carefully design the hierarchical reasoning layers to decouple fully connected multi-head attention and remove the redundant interfering correlations. Furthermore, our proposed pyramid and shifted alignment mechanisms are effective to improve the cross-modal information utilization of neighborhood spatial regions and temporal frames. We conducted extensive experiments to show our HERO outperforms existing techniques by achieving significant improvement on two benchmark datasets.

Motivation & Objective

• To address incomplete cross-modal semantic correspondence in video object grounding, where video content often contains multiple actions not fully aligned with a single-action textual query.
• To overcome the transformer's insensitivity to spatio-temporal locality in video modeling, which limits fine-grained feature alignment.
• To improve cross-modal alignment and feature representation by explicitly modeling spatial and temporal dependencies at multiple granularities.
• To develop a unified, end-to-end framework that enhances both retrieval and grounding accuracy through structured hierarchical reasoning.

Proposed method

• Introduces a weakly-supervised contrastive learning module that classifies video frames as action-consistent or action-independent based on video-caption action semantic correspondence.
• Designs a hierarchical multi-head self-attention mechanism that decouples fully connected attention into attribute-spatial, action-spatial, and action-temporal attention heads to reduce interference and improve locality sensitivity.
• Employs a pyramid alignment mechanism to enhance cross-modal information utilization across neighboring spatial regions and temporal frames.
• Incorporates a shifted alignment mechanism to better capture spatial and temporal dependencies by reweighting attention maps with shifted positions.
• Combines these components into a unified end-to-end model that jointly optimizes action retrieval and object grounding.
• Uses a coarse-to-fine retrieval strategy to first identify action-consistent frames before refining object localization.

Experimental results

Research questions

• RQ1 How can weakly-supervised contrastive learning improve cross-modal alignment in video object grounding when video semantics are finer-grained than the textual query?
• RQ2 To what extent does hierarchical attention with decoupled attention heads improve modeling of spatio-temporal locality in video transformers?
• RQ3 Can pyramid and shifted alignment mechanisms enhance cross-modal feature utilization across spatial and temporal dimensions?
• RQ4 How do the individual components of the HERO framework contribute to overall performance gain in video object grounding?
• RQ5 Does the proposed hierarchical reasoning framework outperform standard transformer-based models on standard benchmarks?

Key findings

• HERO achieves a 63.8% average mAP on the VidSTG dataset, a significant improvement over the previous SOTA method (63.1%) and the base model (60.4%).
• On the MSVD dataset, HERO achieves 64.5% mAP, outperforming the best baseline (63.3%) and demonstrating consistent gains across both benchmarks.
• Ablation studies show that removing any component—such as the coarse-to-fine action retrieval or hierarchical MSA modules—leads to a measurable drop in performance, confirming their necessity.
• The combination of all components yields the highest performance, indicating complementary contributions from contrastive learning, hierarchical attention, and alignment mechanisms.
• Visualization results confirm that HERO predicts more accurate bounding boxes and better aligns with ground truth, especially in complex scenes with multiple actions.
• The model shows stronger representation learning and detection capability compared to STVGBert, the best-performing baseline, particularly in localizing objects described by fine-grained actions.