[Paper Review] Look Closer to Ground Better: Weakly-Supervised Temporal Grounding of Sentence in Video

Zhenfang Chen, Lin Ma+3 more|arXiv (Cornell University)|Jan 25, 2020

Multimodal Machine Learning Applications26 references48 citations

TL;DR

The paper proposes a two-stage weakly-supervised temporal grounding method that first selects a coarse video segment via multi-scale sliding windows and MIL, then refines to a precise frame boundary through fine-grained frame-sentence interaction and watershed-based grouping.

ABSTRACT

In this paper, we study the problem of weakly-supervised temporal grounding of sentence in video. Specifically, given an untrimmed video and a query sentence, our goal is to localize a temporal segment in the video that semantically corresponds to the query sentence, with no reliance on any temporal annotation during training. We propose a two-stage model to tackle this problem in a coarse-to-fine manner. In the coarse stage, we first generate a set of fixed-length temporal proposals using multi-scale sliding windows, and match their visual features against the sentence features to identify the best-matched proposal as a coarse grounding result. In the fine stage, we perform a fine-grained matching between the visual features of the frames in the best-matched proposal and the sentence features to locate the precise frame boundary of the fine grounding result. Comprehensive experiments on the ActivityNet Captions dataset and the Charades-STA dataset demonstrate that our two-stage model achieves compelling performance.

Motivation & Objective

• Motivate reducing dependency on costly temporal annotations for grounding sentences in videos.
• Localize the video segment semantically matching a query without temporal annotations during training.
• Develop a coarse-to-fine framework to achieve precise start/end timestamps.
• Leverage MIL to learn from video-sentence pairs and sliding-window proposals.
• Evaluate on ActivityNet Captions and Charades-STA to demonstrate effectiveness.

Proposed method

• Encode sentence with a Bi-LSTM after GloVe embeddings.
• Encode video with frame features plus Bi-LSTM for contextualization.
• Generate fixed-length temporal proposals via multi-scale sliding windows with 80% overlap.
• Coarse stage: compute fused multimodal scores using a two-stream grounder (classification + selection) and MIL training.
• Fine stage: expand the coarse segment, perform frame-level interactions with the sentence, and predict framewise scores; apply watershed-based grouping to obtain precise boundaries.
• Train in two stages: first coarse stage with MIL loss, then fine stage with rank-based loss to separate correct from incorrect video-sentence pairs.

Experimental results

Research questions

• RQ1 Can weak supervision (no temporal annotations) achieve competitive temporal grounding performance?
• RQ2 Does a coarse-to-fine strategy improve boundary precision over a single-stage approach?
• RQ3 How do frame-level fine-grained interactions influence grounding accuracy compared to proposal-level (coarse) reasoning?

Key findings

• The proposed two-stage model achieves competitive results on ActivityNet Captions and Charades-STA under weak supervision.
• Coarse stage alone significantly outperforms random proposal selection and provides a solid grounding foundation.
• Fine-stage frame-level interactions plus watershed-based grouping improve temporal boundary precision over the coarse result.
• The full coarse-to-fine model outperforms baselines and several weakly-supervised methods, approaching or surpassing some fully-supervised methods on key metrics.
• Ablations show that using the two-stream coarse grounder benefits the coarse stage, while a FC-based fine grounder yields better fine-grained localization.