[Paper Review] End-to-end Multi-modal Video Temporal Grounding

Yiwen Chen, Yi‐Hsuan Tsai+1 more|arXiv (Cornell University)|Jul 12, 2021

Multimodal Machine Learning Applications34 references21 citations

TL;DR

This paper proposes DRFT, an end-to-end multi-modal framework for video temporal grounding that fuses RGB, optical flow, and depth features using a co-attentional transformer for dynamic inter-modal fusion and self-supervised contrastive learning for intra-modal representation enhancement. The method achieves state-of-the-art performance on Charades-STA and ActivityNet Captions by leveraging complementary visual cues from appearance, motion, and structure.

ABSTRACT

We address the problem of text-guided video temporal grounding, which aims to identify the time interval of a certain event based on a natural language description. Different from most existing methods that only consider RGB images as visual features, we propose a multi-modal framework to extract complementary information from videos. Specifically, we adopt RGB images for appearance, optical flow for motion, and depth maps for image structure. While RGB images provide abundant visual cues of certain events, the performance may be affected by background clutters. Therefore, we use optical flow to focus on large motion and depth maps to infer the scene configuration when the action is related to objects recognizable with their shapes. To integrate the three modalities more effectively and enable inter-modal learning, we design a dynamic fusion scheme with transformers to model the interactions between modalities. Furthermore, we apply intra-modal self-supervised learning to enhance feature representations across videos for each modality, which also facilitates multi-modal learning. We conduct extensive experiments on the Charades-STA and ActivityNet Captions datasets, and show that the proposed method performs favorably against state-of-the-art approaches.

Motivation & Objective

• To address the limitations of single-modality video grounding methods that rely solely on RGB features, especially in complex or cluttered backgrounds.
• To improve temporal grounding accuracy by integrating complementary visual cues from multiple modalities: RGB for appearance, optical flow for motion, and depth for scene structure.
• To enable effective inter-modal feature learning through a dynamic fusion mechanism that adapts to input-specific modality importance.
• To enhance feature robustness across videos by applying self-supervised contrastive learning within each modality to learn action-invariant representations.

Proposed method

• The DRFT framework integrates three visual modalities: RGB images for appearance, optical flow for motion, and depth maps for structural context.
• A co-attentional transformer module dynamically fuses features across modalities by modeling cross-attention interactions based on input content.
• The inter-modal fusion mechanism learns modality-specific attention weights, allowing the model to emphasize relevant cues—e.g., flow for high-motion actions, depth for structural actions—depending on the query.
• Intra-modal self-supervised contrastive learning is applied independently to each modality to improve feature consistency across video clips with the same action, regardless of background or context.
• The model is trained end-to-end using a temporal grounding loss that regresses the start and end times of the query-mentioned event.
• The framework is evaluated on Charades-STA and ActivityNet Captions, with ablation studies validating the contribution of each component.

Experimental results

Research questions

• RQ1 How do complementary visual modalities—RGB, optical flow, and depth—improve video temporal grounding compared to using RGB alone?
• RQ2 To what extent does a dynamic, attention-based fusion mechanism enhance inter-modal feature learning in multi-modal video grounding?
• RQ3 Can self-supervised contrastive learning within each modality improve feature representations and boost multi-modal performance?
• RQ4 Which modality (RGB, flow, or depth) contributes most to grounding specific types of actions, such as those with small motion or strong structural cues?
• RQ5 How does the DRFT framework compare to state-of-the-art methods on standard benchmarks like Charades-STA and ActivityNet Captions?

Key findings

• The three-stream DRFT model with RGB, flow, and depth outperforms single-modality baselines on both Charades-STA and ActivityNet Captions, demonstrating the effectiveness of multi-modal fusion.
• The single-stream DRFT models with contrastive learning significantly improve over their non-contrastive counterparts, confirming the benefit of intra-modal self-supervision.
• For actions with small motion like 'smiling', RGB features contribute most, as shown by higher attention weights in the fusion mechanism.
• For high-motion actions like 'closing a door' or 'throwing a pillow', optical flow dominates the fusion process, with the highest attention weights assigned to flow-modulated features.
• For structure-dependent actions like 'sitting in a bed' or 'working at a table', depth maps contribute most, with the highest attention weights on depth-conditioned features.
• Ablation studies confirm that both the inter-modal fusion and intra-modal contrastive learning components are essential for achieving state-of-the-art performance.