[Paper Review] G2L: Semantically Aligned and Uniform Video Grounding via Geodesic and Game Theory

Hongxiang Li, Meng Cao+4 more|arXiv (Cornell University)|Jul 26, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

G2L proposes a novel video grounding framework that improves semantically aligned and uniform representation learning by leveraging geodesic distance to model temporal and semantic correlations among video moments, and game-theoretic Shapley interactions to resolve ambiguities between similar moments. The method outperforms vanilla contrastive learning on three benchmarks by mitigating issues from semantic overlapping and sparse annotations.

ABSTRACT

The recent video grounding works attempt to introduce vanilla contrastive learning into video grounding. However, we claim that this naive solution is suboptimal. Contrastive learning requires two key properties: (1) \emph{alignment} of features of similar samples, and (2) \emph{uniformity} of the induced distribution of the normalized features on the hypersphere. Due to two annoying issues in video grounding: (1) the co-existence of some visual entities in both ground truth and other moments, \ie semantic overlapping; (2) only a few moments in the video are annotated, \ie sparse annotation dilemma, vanilla contrastive learning is unable to model the correlations between temporally distant moments and learned inconsistent video representations. Both characteristics lead to vanilla contrastive learning being unsuitable for video grounding. In this paper, we introduce Geodesic and Game Localization (G2L), a semantically aligned and uniform video grounding framework via geodesic and game theory. We quantify the correlations among moments leveraging the geodesic distance that guides the model to learn the correct cross-modal representations. Furthermore, from the novel perspective of game theory, we propose semantic Shapley interaction based on geodesic distance sampling to learn fine-grained semantic alignment in similar moments. Experiments on three benchmarks demonstrate the effectiveness of our method.

Motivation & Objective

• Address the limitations of vanilla contrastive learning in video grounding, where strict temporal-based positive/negative sampling fails to capture semantic similarities between non-adjacent or overlapping moments.
• Overcome the semantic overlapping issue, where visually similar entities appear in both ground-truth and non-ground-truth moments, leading to inconsistent feature representations.
• Mitigate the sparse annotation dilemma, where unannotated moments are incorrectly pushed apart from relevant queries due to lack of supervision.
• Improve representation uniformity and alignment by modeling cross-modal and cross-moment relationships through geodesic distances on a moment graph.
• Introduce fine-grained semantic distinction between highly similar moments using a game-theoretic Shapley interaction module based on geodesic sampling.

Proposed method

• Construct a moment graph from video clips to model the manifold structure of video representations, enabling geodesic distance computation as a proxy for semantic similarity.
• Replace standard contrastive learning's strict temporal-based positive/negative sampling with geodesic-based sampling, where closer moments on the graph are treated as more semantically aligned.
• Formulate video moments and queries as players in a cooperative game, and compute semantic Shapley interactions to quantify marginal contributions of each component for improved fine-grained alignment.
• Design a differentiable semantic Shapley interaction module that samples intra-video moments based on geodesic distance to focus on nuanced differences between semantically similar clips.
• Integrate geodesic-guided contrastive learning (GCL) and semantic Shapley interaction (SSI) modules into a unified training objective to jointly optimize alignment and uniformity.
• Use t-SNE visualization to demonstrate that G2L reduces representation clustering artifacts (e.g., 'islands') caused by naive contrastive learning, resulting in more uniform and semantically coherent feature spaces.

Experimental results

Research questions

• RQ1 Can geodesic distance on a moment graph better capture semantic correlations between temporally distant or overlapping video moments than standard temporal or Euclidean distance?
• RQ2 How does replacing strict temporal-based contrastive learning with geodesic-based sampling affect the alignment and uniformity of video-text representations?
• RQ3 To what extent can game-theoretic Shapley interactions improve discrimination between semantically similar video moments in video grounding?
• RQ4 Does the proposed G2L framework effectively mitigate the negative impacts of semantic overlapping and sparse annotation in video grounding?
• RQ5 Can the integration of geodesic distance and Shapley interaction lead to consistent performance gains across multiple video grounding benchmarks?

Key findings

• G2L achieves state-of-the-art performance on three video grounding benchmarks—ActivityNet-Captions, MSVD, and MSVD-ASR—demonstrating consistent improvements over baseline contrastive learning methods.
• Using geodesic distance instead of Euclidean, timestamp, or cosine distance improves mAP by up to 5% (Euclidean), 2% (timestamp), and 3% (cosine), showing its superiority in capturing semantic structure.
• The semantic Shapley interaction module reduces confusion between visually similar moments, as evidenced by t-SNE visualization showing elimination of 'island' clusters in feature space.
• G2L reduces the performance drop caused by sparse annotations by modeling relationships between unannotated moments and queries through geodesic paths.
• Inference time increases by only 3 seconds compared to the base model, indicating that the G2L framework is efficient despite additional training complexity from K-NN graph and SSI modules.
• Qualitative results show that G2L successfully grounds queries referring to events that reappear later in the video (e.g., 'knitting again', 'the second belt'), which standard contrastive methods fail to localize.