[Paper Review] Knowing Where to Focus: Event-aware Transformer for Video Grounding

Jinhyun Jang, Jungin Park+3 more|arXiv (Cornell University)|Aug 14, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

This paper proposes EaTR, a novel Event-aware Transformer for video grounding that formulates dynamic, video-specific moment queries using event reasoning via slot attention and gated fusion to enhance temporal localization. By learning event units as content- and position-aware queries, EaTR achieves state-of-the-art performance with faster convergence and improved accuracy on multiple benchmarks.

ABSTRACT

Recent DETR-based video grounding models have made the model directly predict moment timestamps without any hand-crafted components, such as a pre-defined proposal or non-maximum suppression, by learning moment queries. However, their input-agnostic moment queries inevitably overlook an intrinsic temporal structure of a video, providing limited positional information. In this paper, we formulate an event-aware dynamic moment query to enable the model to take the input-specific content and positional information of the video into account. To this end, we present two levels of reasoning: 1) Event reasoning that captures distinctive event units constituting a given video using a slot attention mechanism; and 2) moment reasoning that fuses the moment queries with a given sentence through a gated fusion transformer layer and learns interactions between the moment queries and video-sentence representations to predict moment timestamps. Extensive experiments demonstrate the effectiveness and efficiency of the event-aware dynamic moment queries, outperforming state-of-the-art approaches on several video grounding benchmarks.

Motivation & Objective

• Address the limitation of input-agnostic moment queries in DETR-based video grounding models, which lack awareness of video-specific temporal structures.
• Improve temporal reasoning by learning video-specific event units that serve as dynamic moment queries with content and positional awareness.
• Enhance interaction between sentence and video representations by filtering out irrelevant moment queries and emphasizing sentence-relevant ones.
• Achieve faster training convergence and higher accuracy by providing reliable, input-specific referential search areas.
• Propose a unified framework that integrates event-level and moment-level reasoning for more precise video grounding.

Proposed method

• Use a slot attention mechanism to identify distinctive event units from video features, producing content and positional queries as initial dynamic moment queries.
• Introduce a gated fusion transformer (GF) layer that adaptively fuses moment queries with sentence representations based on similarity, enhancing informative queries and suppressing irrelevant ones.
• Construct dynamic moment queries from identified event units, which serve as input-specific referential search areas during inference.
• Employ a transformer decoder with cross-attention to allow moment queries to interact with video-sentence representations, enabling precise timestamp prediction.
• Train the model with a multi-task objective including a novel event localization loss ($\mathcal{L}_{\text{event}}$) and standard detection loss, balanced via hyperparameter $\lambda_{\text{event}}$.
• Use learnable query embeddings for moment queries, initialized via event units, and update them end-to-end during training.

(a) Video grounding

Experimental results

Research questions

• RQ1 Can event-aware dynamic moment queries improve video grounding performance by incorporating video-specific temporal structure?
• RQ2 How does the gated fusion mechanism enhance the relevance of moment queries to the input sentence compared to fixed or unweighted query interactions?
• RQ3 Does learning event units via slot attention lead to faster training convergence and better generalization than input-agnostic moment queries?
• RQ4 What is the optimal balance between the event localization loss and detection loss in the training objective?
• RQ5 How do dynamic moment queries compare to fixed input-agnostic queries in terms of attention distribution and prediction reliability?

Key findings

• EaTR achieves state-of-the-art performance on multiple video grounding benchmarks, including ActivityNet-1.3, TACRED, and MSVD, outperforming previous SOTA methods such as Moment-DETR.
• The model converges significantly faster than Moment-DETR, demonstrating that dynamic, event-aware queries improve training efficiency by providing better initial referential search areas.
• Ablation studies show that the gated fusion layer outperforms addition, concatenation, and multi-head cross-attention fusion, confirming its effectiveness in filtering irrelevant queries.
• The optimal number of event slots $N$ is 10, as performance degrades with larger or smaller values due to difficulty in capturing long events or overfitting.
• The balancing hyperparameter $\lambda_{\text{event}}$ has a strong impact: performance peaks at $\lambda_{\text{event}} = 3$, with lower or higher values leading to degradation.
• Attention visualization confirms that EaTR’s dynamic queries adapt to video content and provide balanced, informative attention patterns, while input-agnostic queries rely too heavily on video-sentence interaction, leading to errors.

(b) Previous DETR-based approach