[Paper Review] Multi-Scale Self-Contrastive Learning with Hard Negative Mining for Weakly-Supervised Query-based Video Grounding

Shentong Mo, Daizong Liu+1 more|arXiv (Cornell University)|Mar 8, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes a multi-scale self-contrastive learning framework with hard negative mining for weakly-supervised query-based video grounding, enabling frame-wise matching score prediction without redundant segment proposals. By iteratively mining multi-scale hard negatives in a coarse-to-fine manner, the method learns discriminative frame-level representations, achieving state-of-the-art performance on two benchmarks with only video-level supervision.

ABSTRACT

Query-based video grounding is an important yet challenging task in video understanding, which aims to localize the target segment in an untrimmed video according to a sentence query. Most previous works achieve significant progress by addressing this task in a fully-supervised manner with segment-level labels, which require high labeling cost. Although some recent efforts develop weakly-supervised methods that only need the video-level knowledge, they generally match multiple pre-defined segment proposals with query and select the best one, which lacks fine-grained frame-level details for distinguishing frames with high repeatability and similarity within the entire video. To alleviate the above limitations, we propose a self-contrastive learning framework to address the query-based video grounding task under a weakly-supervised setting. Firstly, instead of utilizing redundant segment proposals, we propose a new grounding scheme that learns frame-wise matching scores referring to the query semantic to predict the possible foreground frames by only using the video-level annotations. Secondly, since some predicted frames (i.e., boundary frames) are relatively coarse and exhibit similar appearance to their adjacent frames, we propose a coarse-to-fine contrastive learning paradigm to learn more discriminative frame-wise representations for distinguishing the false positive frames. In particular, we iteratively explore multi-scale hard negative samples that are close to positive samples in the representation space for distinguishing fine-grained frame-wise details, thus enforcing more accurate segment grounding. Extensive experiments on two challenging benchmarks demonstrate the superiority of our proposed method compared with the state-of-the-art methods.

Motivation & Objective

• Address the high annotation cost of fully-supervised query-based video grounding by leveraging only video-level supervision.
• Overcome limitations of prior weakly-supervised methods that rely on redundant, pre-defined segment proposals with poor frame-level discrimination.
• Improve fine-grained frame-wise representation learning to distinguish hard negative frames near ground-truth boundaries.
• Develop a coarse-to-fine contrastive learning strategy that progressively mines harder negative samples in both local frame and global segment scales.

Proposed method

• Propose frame-wise matching score prediction using an attention mechanism to estimate foreground likelihood per frame, replacing reliance on segment proposals.
• Introduce a multi-scale self-contrastive learning paradigm that dynamically defines positive and negative samples based on predicted scores.
• Implement a coarse-to-fine iterative hard negative mining strategy, where the sampling range is progressively updated to target increasingly similar negative frames.
• Explore hard negatives at both local frame-scale and non-local segment-scale to learn multi-scale intrinsic features.
• Balance frame-level and segment-level losses using learnable hyperparameters (λ_fra and λ_seg) to optimize grounding accuracy.
• Use a learnable threshold to group consecutive high-scoring frames into the final predicted segment.

Experimental results

Research questions

• RQ1 Can frame-wise matching score prediction replace redundant segment proposals in weakly-supervised video grounding?
• RQ2 How can self-contrastive learning with iterative hard negative mining improve frame-level representation discrimination in the absence of frame-level annotations?
• RQ3 What is the impact of multi-scale (frame and segment) hard negative sampling on distinguishing similar foreground and background frames?
• RQ4 How does the progressive mining of increasingly hard negatives affect grounding performance compared to one-step negative sampling?
• RQ5 What is the optimal balance between frame-level and segment-level loss components in the training objective?

Key findings

• The proposed MSCL framework achieves state-of-the-art performance on both Charades-STA and ActivityNet-Caption benchmarks under weakly-supervised settings.
• The model outperforms prior methods such as SCN and VGN+CCL, especially in accurately localizing segment boundaries due to improved discrimination of hard negative frames.
• The optimal hyperparameter setting is λ_fra = 10 and λ_seg = 5, indicating that balancing frame-level and segment-level supervision is critical for performance.
• The model achieves peak performance at a batch size of 16 and shows robustness to batch size variations, unlike prior contrastive learning methods requiring large batches.
• Visualization shows that the iterative hard negative mining process progressively selects samples closer to the ground-truth segment, confirming the effectiveness of the dynamic sampling strategy.
• Frame-wise score curves demonstrate that the model predicts the highest score above the upper bound $b_u$ for the correct segment, validating accurate segment localization.