[Paper Review] MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformer

Yifang Xu, Yunzhuo Sun+4 more|arXiv (Cornell University)|Apr 29, 2023

Visual Attention and Saliency Detection4 citations

TL;DR

MH-DETR proposes a novel end-to-end transformer-based model for joint video moment and highlight detection, introducing a pooling-based token mixer for intra-modal context and a plug-and-play cross-modal interaction module to enhance feature alignment. It achieves state-of-the-art performance across multiple benchmarks, including QVHighlights, Charades-STA, ActivityNet, and TVSum, outperforming prior methods by significant margins while training from scratch without pre-training.

ABSTRACT

With the increasing demand for video understanding, video moment and highlight detection (MHD) has emerged as a critical research topic. MHD aims to localize all moments and predict clip-wise saliency scores simultaneously. Despite progress made by existing DETR-based methods, we observe that these methods coarsely fuse features from different modalities, which weakens the temporal intra-modal context and results in insufficient cross-modal interaction. To address this issue, we propose MH-DETR (Moment and Highlight Detection Transformer) tailored for MHD. Specifically, we introduce a simple yet efficient pooling operator within the uni-modal encoder to capture global intra-modal context. Moreover, to obtain temporally aligned cross-modal features, we design a plug-and-play cross-modal interaction module between the encoder and decoder, seamlessly integrating visual and textual features. Comprehensive experiments on QVHighlights, Charades-STA, Activity-Net, and TVSum datasets show that MH-DETR outperforms existing state-of-the-art methods, demonstrating its effectiveness and superiority. Our code is available at https://github.com/YoucanBaby/MH-DETR.

Motivation & Objective

• To address the limitations of existing DETR-based methods in video moment and highlight detection (MHD), which suffer from coarse cross-modal fusion and weak intra-modal context modeling.
• To improve temporal alignment and representation learning between visual and textual modalities in MHD by designing a dedicated cross-modal interaction module.
• To enhance global context modeling within unimodal features using a simple yet effective pooling operator in the uni-modal encoder.
• To achieve state-of-the-art performance on MHD benchmarks without requiring pre-training or post-processing steps like non-maximum suppression.

Proposed method

• Introduces a pooling-based token mixer within the uni-modal encoder to capture global intra-modal context, replacing standard self-attention for efficiency and effectiveness.
• Designs a plug-and-play cross-modal interaction module between the encoder and decoder to enable temporally aligned fusion of visual and textual features.
• Uses a dual-branch transformer encoder-decoder architecture inspired by DETR, with moment decoder predicting start and end timestamps and highlight head regressing clip-wise saliency scores.
• Employs bipartite matching and end-to-end training with joint loss optimization for both moment retrieval and highlight detection.
• Applies a multi-head cross-attention mechanism where visual features are queried with textual features to emphasize semantically relevant visual content.
• Trains the model from scratch using only raw visual and textual features, without additional pre-training or data augmentation.

Experimental results

Research questions

• RQ1 Can a simple pooling-based token mixer effectively model global intra-modal context in video and text representations for MHD?
• RQ2 Does a dedicated cross-modal interaction module between encoder and decoder improve feature alignment and performance in joint moment and highlight detection?
• RQ3 How does the proposed MH-DETR compare to state-of-the-art methods in terms of accuracy and efficiency on multiple MHD benchmarks?
• RQ4 To what extent does co-optimization of moment retrieval and highlight detection losses improve overall performance?
• RQ5 Is the proposed method robust and generalizable across diverse datasets such as QVHighlights, Charades-STA, ActivityNet, and TVSum?

Key findings

• On the QVHighlights dataset, MH-DETR achieves a 1.84% absolute improvement in R1@0.5 and 1.04% in R1@0.7 over Moment-DETR when using VGG features.
• On Charades-STA, MH-DETR outperforms the recent SOTA method VISA by a significant margin, especially with I3D features, demonstrating strong generalization.
• On TVSum, MH-DETR achieves an average top-5 mAP of 85.6%, surpassing SL-Module by 8.3 percentage points across all categories.
• Ablation studies confirm that removing either the highlight loss or moment loss leads to a significant performance drop, validating the importance of joint optimization.
• The cross-modal interaction module contributes most to performance gain, with models including it outperforming those without by up to 3.5% in mAP on QVHighlights.
• MH-DETR achieves SOTA results even when matched for parameter count, outperforming both Moment-DETR-L and MH-DETR-S, indicating superior parameter efficiency.