[論文レビュー] MomentDiff: Generative Video Moment Retrieval from Random to Real

Pandeng Li, Chen-Wei Xie+6 more|arXiv (Cornell University)|Jul 6, 2023

Multimodal Machine Learning Applications被引用数 23

ひとこと要約

MomentDiffは、ランダムなスパンから開始し、それらをテキスト-ビデオの類似性に導かれて反復的に正確な時間的境界へと洗練させる拡散ベースの生成フレームワークをビデオモーメント検索に導入し、データセットの位置バイアスへの依存を減らします。

ABSTRACT

Video moment retrieval pursues an efficient and generalized solution to identify the specific temporal segments within an untrimmed video that correspond to a given language description. To achieve this goal, we provide a generative diffusion-based framework called MomentDiff, which simulates a typical human retrieval process from random browsing to gradual localization. Specifically, we first diffuse the real span to random noise, and learn to denoise the random noise to the original span with the guidance of similarity between text and video. This allows the model to learn a mapping from arbitrary random locations to real moments, enabling the ability to locate segments from random initialization. Once trained, MomentDiff could sample random temporal segments as initial guesses and iteratively refine them to generate an accurate temporal boundary. Different from discriminative works (e.g., based on learnable proposals or queries), MomentDiff with random initialized spans could resist the temporal location biases from datasets. To evaluate the influence of the temporal location biases, we propose two anti-bias datasets with location distribution shifts, named Charades-STA-Len and Charades-STA-Mom. The experimental results demonstrate that our efficient framework consistently outperforms state-of-the-art methods on three public benchmarks, and exhibits better generalization and robustness on the proposed anti-bias datasets. The code, model, and anti-bias evaluation datasets are available at https://github.com/IMCCretrieval/MomentDiff.

研究の動機と目的

• Motivate a generative perspective for Video Moment Retrieval (VMR) to mitigate temporal location biases in datasets.
• Propose MomentDiff, a diffusion-based framework that maps random spans to real moments via text-video similarity guidance.
• Demonstrate efficiency, generalization, and robustness across multiple public datasets and anti-bias benchmarks.
• Provide anti-bias evaluation datasets to study the impact of location distribution shifts on VMR models.

提案手法

• Represent video and text with multimodal embeddings through similarity-aware conditioning.
• Use a similarity loss to encode coarse-grained text-video relations into fusion embeddings that guide generation.
• Introduce a Video Moment Denoiser (VMD) that reverses a diffusion process to transform random spans into ground-truth moments.
• Model forward diffusion by corrupting real spans with Gaussian noise and learn a denoising network that predicts span coordinates and confidence.
• Implement span embedding via a simple linear projection (FC) rather than ROI-based features to avoid proposal bottlenecks.
• Train with a combined loss including L1, IoU, and cross-entropy terms, plus a similarity loss to align multimodal features.

実験結果

リサーチクエスチョン

• RQ1 Can a diffusion-based generator learn to create accurate video moment spans from random initialization without relying on predefined proposals?
• RQ2 Does MomentDiff provide better generalization and robustness to temporal location biases than discriminative VMR methods?
• RQ3 How do anti-bias location-shift datasets (Charades-STA-Len and Charades-STA-Mom) affect VMR performance and model robustness?
• RQ4 What is the impact of design choices (span embedding, diffusion scale, VMD, loss design) on VMR performance?

主な発見

Method	特徴	R1@0.5	R1@0.7	MAP@0.5	MAP@0.75	MAP_avg
MomentDiff	SF+C, Glove	55.57	32.42	61.07	32.51	32.85

• MomentDiff consistently outperforms state-of-the-art methods on Charades-STA, QVHighlights, and TACoS across various feature setups.
• MomentDiff shows stronger robustness and generalization on anti-bias datasets Charades-STA-Len and Charades-STA-Mom compared to MomentDETR.
• With SF+C features and CLIP-based text, MomentDiff achieves top performance on Charades-STA (e.g., R1@0.5 up to 55.57, MAP_avg 32.85).
• Ablation studies show FC span embedding beats ROI-based embedding, diffusion with VMD and noise conditioning improves results, and a balanced set of losses is beneficial.
• Transfer experiments indicate MomentDiff maintains strong performance in cross-domain settings (Charades-CD and ActivityNet-CD) beyond its training distribution.