[Paper Review] Augmented 2D-TAN: A Two-stage Approach for Human-centric Spatio-Temporal Video Grounding

Chaolei Tan, Zihang Lin+3 more|arXiv (Cornell University)|Jun 20, 2021

Multimodal Machine Learning Applications6 references4 citations

TL;DR

This paper proposes Augmented 2D-TAN, a two-stage method for human-centric spatio-temporal video grounding that improves temporal grounding via a Bi-LSTM aggregation module and Random Concatenation Augmentation (RCA), achieving state-of-the-art performance with 31.9% mean viou on the HC-STVG test set.

ABSTRACT

We propose an effective two-stage approach to tackle the problem of language-based Human-centric Spatio-Temporal Video Grounding (HC-STVG) task. In the first stage, we propose an Augmented 2D Temporal Adjacent Network (Augmented 2D-TAN) to temporally ground the target moment corresponding to the given description. Primarily, we improve the original 2D-TAN from two aspects: First, a temporal context-aware Bi-LSTM Aggregation Module is developed to aggregate clip-level representations, replacing the original max-pooling. Second, we propose to employ Random Concatenation Augmentation (RCA) mechanism during the training phase. In the second stage, we use pretrained MDETR model to generate per-frame bounding boxes via language query, and design a set of hand-crafted rules to select the best matching bounding box outputted by MDETR for each frame within the grounded moment.

Motivation & Objective

• Address the challenge of human-centric spatio-temporal video grounding (HC-STVG), where the goal is to localize a person in both time and space using natural language.
• Overcome limitations in existing 2D-TAN models, such as overfitting and non-discriminative moment feature representation due to max-pooling.
• Improve temporal grounding accuracy by enhancing feature aggregation and data augmentation strategies.
• Achieve state-of-the-art performance on the HC-STVG benchmark through a two-stage pipeline combining temporal and spatial grounding.

Proposed method

• Propose a Bi-LSTM Aggregation Module to replace max-pooling in 2D-TAN, enabling better modeling of long-term temporal variations in moment features.
• Design a Random Concatenation Augmentation (RCA) strategy that randomly concatenates two videos and their queries during training to increase data diversity and reduce overfitting.
• Construct a 2D feature map M ∈ ℝ^{N×N×c} where M_ij is computed via the Bi-LSTM aggregation module for valid moment proposals (i ≤ j), and zero otherwise.
• Use a pretrained MDETR model to generate per-frame bounding box proposals with grounding text, followed by rule-based filtering to select the most relevant box.
• Apply a dependency parser to extract the subject from the input sentence and filter MDETR outputs that do not match the subject or contain multiple persons.
• Select the bounding box with the longest grounding text from the filtered set as the final spatial prediction for each frame in the grounded temporal segment.

Experimental results

Research questions

• RQ1 Can a Bi-LSTM-based aggregation module improve the discriminativeness of moment features compared to max-pooling in 2D-TAN?
• RQ2 Does Random Concatenation Augmentation (RCA) effectively reduce overfitting and improve generalization in HC-STVG?
• RQ3 Can a two-stage pipeline combining temporal grounding with per-frame referring improve spatial localization accuracy?
• RQ4 What is the impact of combining Bi-LSTM aggregation and RCA on overall HC-STVG performance?
• RQ5 How does the proposed method compare to existing state-of-the-art approaches on the HC-STVG benchmark?

Key findings

• The Bi-LSTM Aggregation Module improved the viou@0.5 score from 16.7% to 18.8% on the validation split, demonstrating better feature representation.
• Random Concatenation Augmentation (RCA) increased the viou@0.5 score from 16.7% to 17.0% on the validation split, indicating improved generalization.
• The ablation study confirmed that both Bi-LSTM and RCA contribute consistently to performance gains, with the combined method achieving 30.4% viou@0.5 on the validation set.
• The ensemble of 10 Augmented 2D-TAN models achieved 31.9% mean viou on the HC-STVG test set, outperforming all prior methods listed in the leaderboard.
• The proposed method achieved 52.7% viou@0.3 and 56.5% tiou on the test set, confirming its state-of-the-art performance on the HC-STVG benchmark.