[Paper Review] ExCL: Extractive Clip Localization Using Natural Language Descriptions

Soham Ghosh, Anuva Agarwal+2 more|arXiv (Cornell University)|Apr 4, 2019

Video Analysis and Summarization78 citations

TL;DR

ExCL predicts exact start and end frames for a video clip given a natural language query by learning cross-modal interactions, outperforming prior ranking-based methods on TACoS and ActivityNet and matching on Charades-STA. It models three span-predictor variants and uses both classification and regression objectives.

ABSTRACT

The task of retrieving clips within videos based on a given natural language query requires cross-modal reasoning over multiple frames. Prior approaches such as sliding window classifiers are inefficient, while text-clip similarity driven ranking-based approaches such as segment proposal networks are far more complicated. In order to select the most relevant video clip corresponding to the given text description, we propose a novel extractive approach that predicts the start and end frames by leveraging cross-modal interactions between the text and video - this removes the need to retrieve and re-rank multiple proposal segments. Using recurrent networks we encode the two modalities into a joint representation which is then used in different variants of start-end frame predictor networks. Through extensive experimentation and ablative analysis, we demonstrate that our simple and elegant approach significantly outperforms state of the art on two datasets and has comparable performance on a third.

Motivation & Objective

• Motivate extractive clip localization over ranking-based approaches that rely on fixed candidate clips.
• Propose a modular cross-modal framework that directly predicts start and end frames from text-video interactions.
• Evaluate different span-predictor architectures and training objectives across diverse datasets.
• Show that an extractive model with temporal context yields strong performance and generalizes across datasets.

Proposed method

• Encode text with a bidirectional LSTM using GloVe embeddings to obtain a sentence embedding.
• Encode video with a bidirectional LSTM over I3D features to capture temporal context.
• Compute per-frame start/end scores with three span-predictor variants: MLP, Tied-LSTM, and Conditioned-LSTM.
• Train with either classification loss (softmax-normalized start/end probabilities) or regression loss (expectation over softmax distributions).
• For regression, model P(end|start) via SoftMax over masked logits to enforce end >= start and use expected start/end times as predictions.

Experimental results

Research questions

• RQ1 Can an extractive, end-to-end model locate the exact video clip described by a natural language query without ranking multiple proposals?
• RQ2 How do different cross-modal span-predictor architectures affect localization accuracy across diverse datasets?
• RQ3 Is a regression objective advantageous over a classification objective for precise temporal localization?
• RQ4 How does the model perform across datasets with varying video lengths and vocabulary?
• RQ5 What is the impact of including a video LSTM encoder on performance?

Key findings

• The extractive model significantly outperforms prior ranking-based baselines on TACoS and ActivityNet.
• Incorporating a video LSTM substantially improves performance, and span-predictors with recurrent encoding (especially tied LSTM) yield strong results across datasets.
• Regression training offers comparable or slightly better results than classification, with no clear loss in information.
• Without a video LSTM, recurrent span predictors are essential to capture cross-modal interactions.
• Tied-LSTM span predictor generally provides the best or near-best results across datasets and settings.
• Characteristically, TACoS remains the most challenging benchmark due to high temporal precision requirements.