[Paper Review] Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language

Songyang Zhang, Houwen Peng+2 more|arXiv (Cornell University)|Dec 8, 2019

Multimodal Machine Learning Applications36 references42 citations

TL;DR

The paper introduces a 2D Temporal Adjacent Network (2D-TAN) that localizes moments by modeling temporal relations on a 2D start-end map, achieving state-of-the-art results on Charades-STA, ActivityNet Captions, and TACoS without extra post-processing.

ABSTRACT

We address the problem of retrieving a specific moment from an untrimmed video by a query sentence. This is a challenging problem because a target moment may take place in relations to other temporal moments in the untrimmed video. Existing methods cannot tackle this challenge well since they consider temporal moments individually and neglect the temporal dependencies. In this paper, we model the temporal relations between video moments by a two-dimensional map, where one dimension indicates the starting time of a moment and the other indicates the end time. This 2D temporal map can cover diverse video moments with different lengths, while representing their adjacent relations. Based on the 2D map, we propose a Temporal Adjacent Network (2D-TAN), a single-shot framework for moment localization. It is capable of encoding the adjacent temporal relation, while learning discriminative features for matching video moments with referring expressions. We evaluate the proposed 2D-TAN on three challenging benchmarks, i.e., Charades-STA, ActivityNet Captions, and TACoS, where our 2D-TAN outperforms the state-of-the-art.

Motivation & Objective

• Motivation to localize moments in untrimmed videos described by natural language with improved context modeling.
• Propose a representation that captures adjacent temporal relations via a 2D map of candidate moments.
• Build a single-shot network that directly scores moment candidates using fused video-language features.
• Demonstrate that explicit context modeling across moment candidates improves localization accuracy.

Proposed method

• Represent the query with a 3-layer LSTM over GloVe embeddings to obtain f^S.
• Extract 2D temporal feature map F^M by pooling across clips to form start-end indices (a,b) with features f^V, yielding F^M in R^{N x N x d^V}.
• Use a sparse sampling strategy to select a compact set of moment candidates, governed by G(a,b) to reduce computation.
• Fuse video-language features by projecting into a common space and applying Hadamard product with Frobenius normalization as F = ||(w^S f^S 1^T) ⊙ (W^M F^M)||_F.
• Model context via an L-layer 2D convolutional Temporal Adjacent Network over F to capture adjacent moment relations.
• Train with a scaled IoU supervision y_i derived from ground-truth IoU o_i, using a binary cross-entropy loss over C valid candidates.

Experimental results

Research questions

• RQ1 Can a 2D temporal map effectively model dependencies among candidate moments for natural language grounded localization?
• RQ2 Does explicit contextual modeling of adjacent moments improve discrimination among visually similar candidates?
• RQ3 How does the proposed sparse candidate sampling affect performance and efficiency compared with dense enumeration?

Key findings

• 2D-TAN achieves competitive to state-of-the-art performance on Charades-STA, ActivityNet Captions, and TACoS across multiple metrics.
• On TACoS, 2D-TAN surpasses ACL-K and TGN by more than 5 and 14 points in Rank1@0.5 and Rank5@0.5, respectively.
• On the large-scale ActivityNet Captions dataset, 2D-TAN outperforms the top method CMIN in IoU@0.5 and IoU@0.7.
• A larger receptive field via deeper or larger kernels improves performance, with saturation when the field becomes sufficiently large.
• Sparse sampling reduces candidates while maintaining comparable performance to dense enumeration, enhancing efficiency.
• Compared to sliding-window methods, 2D-TAN benefits from modeling temporal dependencies among moment candidates rather than treating them independently.