[Paper Review] Multi-Scale 2D Temporal Adjacent Networks for Moment Localization with Natural Language

Songyang Zhang, Houwen Peng+3 more|arXiv (Cornell University)|Dec 4, 2020

Multimodal Machine Learning Applications62 references4 citations

TL;DR

This paper proposes Multi-Scale 2D Temporal Adjacency Networks (MS-2D-TAN), a single-shot framework for moment localization in untrimmed videos using natural language queries. By modeling temporal context through multi-scale 2D maps where one axis represents start time and the other duration, the method captures adjacent moment relationships, enabling improved localization accuracy over state-of-the-art methods on Charades-STA, ActivityNet Captions, and TACoS benchmarks.

ABSTRACT

We address the problem of retrieving a specific moment from an untrimmed video by natural language. It is a challenging problem because a target moment may take place in the context of other temporal moments in the untrimmed video. Existing methods cannot tackle this challenge well since they do not fully consider the temporal contexts between temporal moments. In this paper, we model the temporal context between video moments by a set of predefined two-dimensional maps under different temporal scales. For each map, one dimension indicates the starting time of a moment and the other indicates the duration. These 2D temporal maps can cover diverse video moments with different lengths, while representing their adjacent contexts at different temporal scales. Based on the 2D temporal maps, we propose a Multi-Scale Temporal Adjacent Network (MS-2D-TAN), a single-shot framework for moment localization. It is capable of encoding the adjacent temporal contexts at each scale, while learning discriminative features for matching video moments with referring expressions. We evaluate the proposed MS-2D-TAN on three challenging benchmarks, i.e., Charades-STA, ActivityNet Captions, and TACoS, where our MS-2D-TAN outperforms the state of the art.

Motivation & Objective

• To address the challenge of localizing specific video moments based on natural language queries, especially when context from neighboring moments is essential.
• To overcome limitations of existing two-stage methods that treat moment candidates in isolation, neglecting their temporal relationships.
• To improve localization accuracy by modeling adjacent temporal contexts through structured 2D maps at multiple temporal scales.
• To reduce computational cost while preserving fine-grained localization capability through sparse multi-scale sampling of moment candidates.
• To enable discriminative feature learning that distinguishes semantically similar but visually overlapping moments.

Proposed method

• Constructing a 2D temporal map where each cell represents a moment defined by its start time (on one axis) and duration (on the other), with time unit τ determining localization granularity.
• Creating multiple 2D maps with different time units (e.g., τ, 2τ, 4τ) to form a sparse multi-scale representation, reducing computational cost from O(N²) to O(N).
• Using gated convolutional layers to encode features across the 2D map, allowing the network to attend to adjacent moments and learn contextual representations.
• Employing a single-shot inference strategy that directly predicts moment scores on the 2D map, avoiding the need for proposal generation and matching pipelines.
• Integrating multi-scale maps to increase the receptive field and capture diverse temporal ranges, enhancing robustness to varying moment lengths.
• Applying a sliding window strategy during training with variable window sizes (N) to ensure sufficient context is observed for effective feature learning.

Experimental results

Research questions

• RQ1 Can modeling temporal context between video moments through a 2D map improve moment localization accuracy in natural language queries?
• RQ2 Does multi-scale 2D representation reduce computational cost while maintaining or improving localization performance?
• RQ3 Can a single-shot framework based on 2D maps outperform two-stage methods that process moment candidates independently?
• RQ4 How does the receptive field size and kernel configuration affect the model’s ability to distinguish semantically similar moments?
• RQ5 To what extent does the number of moment candidates influence performance, and can a smaller set of anchors still achieve superior results?

Key findings

• MS-2D-TAN achieves state-of-the-art performance on three benchmarks: Charades-STA, ActivityNet Captions, and TACoS, outperforming prior methods.
• The model achieves a 5.8% higher average mIoU on TACoS compared to the previous SOTA method CBP, despite using approximately half the number of anchors.
• Using a larger sliding window size (N=512 clips) covering ~96.42% of average video length leads to better performance than smaller windows covering 67.47% or 87.57%.
• Increasing the hidden state size improves performance, with a model using 2/3 the parameters of CBP achieving superior results.
• The use of larger kernel sizes with fewer layers (e.g., kernel size 7 with 2 layers) yields better performance than smaller kernels with more layers, given the same receptive field.
• Qualitative analysis shows that the model predicts high scores for both the ground-truth moment and its semantically similar neighbors, indicating effective discriminative feature learning from context.