[Paper Review] Siamese Tracking with Lingual Object Constraints

Maximilian Filtenborg, Efstratios Gavves+1 more|arXiv (Cornell University)|Nov 23, 2020

Video Surveillance and Tracking Methods26 references4 citations

TL;DR

This paper introduces a novel visual tracking framework that incorporates natural language constraints—such as 'near a yellow car'—to filter and localize target objects in videos. By extending SiamRPN++ with vision-language attention mechanisms and a Pyramid Pooling Module, the proposed SiamCT-CA model achieves state-of-the-art performance on two new benchmark datasets, c-MOT16 and c-LaSOT, enabling selective video compression based on constraint validity.

ABSTRACT

Classically, visual object tracking involves following a target object throughout a given video, and it provides us the motion trajectory of the object. However, for many practical applications, this output is often insufficient since additional semantic information is required to act on the video material. Example applications of this are surveillance and target-specific video summarization, where the target needs to be monitored with respect to certain predefined constraints, e.g., 'when standing near a yellow car'. This paper explores, tracking visual objects subjected to additional lingual constraints. Differently from Li et al., we impose additional lingual constraints upon tracking, which enables new applications of tracking. Whereas in their work the goal is to improve and extend upon tracking itself. To perform benchmarks and experiments, we contribute two datasets: c-MOT16 and c-LaSOT, curated through appending additional constraints to the frames of the original LaSOT and MOT16 datasets. We also experiment with two deep models SiamCT-DFG and SiamCT-CA, obtained through extending a recent state-of-the-art Siamese tracking method and adding modules inspired from the fields of natural language processing and visual question answering. Through experimental results, we show that the proposed model SiamCT-CA can significantly outperform its counterparts. Furthermore, our method enables the selective compression of videos, based on the validity of the constraint.

Motivation & Objective

• To address the limitation of classical visual tracking, which only provides motion trajectories without semantic context, by incorporating user-defined natural language constraints.
• To enable tracking systems to selectively identify frames where the target satisfies specific linguistic conditions, such as proximity to an object or action.
• To develop a new benchmarking framework for constraint-based tracking by extending LaSOT and MOT16 with natural language annotations.
• To design and evaluate deep learning models that jointly reason over visual appearance and linguistic descriptions for improved tracking under constraints.
• To demonstrate the feasibility of using constraint-aware tracking for applications like surveillance and video summarization, including selective video compression.

Proposed method

• Extends the SiamRPN++ architecture with a multi-head self-attention mechanism inspired by Visual Question Answering (VQA), enabling joint reasoning over image and text modalities.
• Introduces the SiamCT-CA model, which uses the MCAN architecture for cross-attention between the linguistic constraint and visual features, enhancing alignment between language and visual content.
• Incorporates a Pyramid Pooling Module (PPM) to aggregate multi-scale visual features, improving robustness to scale variations in constrained tracking.
• Employs a dual-branch Siamese network to extract features from the template (initial target frame) and search region (current frame), followed by correlation-based localization.
• Uses a learnable attention mechanism to dynamically attend to relevant words in the constraint and corresponding visual regions in the image, enabling context-aware tracking.
• Trains the model end-to-end using a combination of localization loss and a contrastive loss for constraint grounding, optimizing for both tracking accuracy and constraint compliance.

Experimental results

Research questions

• RQ1 Can visual tracking be enhanced by incorporating natural language constraints that describe the target’s environment or behavior?
• RQ2 How do vision-language attention mechanisms improve the accuracy and robustness of tracking under complex linguistic conditions?
• RQ3 To what extent does the Pyramid Pooling Module contribute to performance gains in multi-scale and long-term tracking under constraints?
• RQ4 Can constraint-aware tracking enable practical applications such as selective video compression or summarization based on semantic criteria?
• RQ5 How do different attention mechanisms (e.g., self-attention vs. self-guided attention) affect the model’s ability to localize targets according to linguistic descriptions?

Key findings

• The SiamCT-CA model significantly outperforms SiamCT-DFG and SiamCT-CA+PPM on both c-MOT16 and c-LaSOT datasets, achieving the highest average precision and AUC scores.
• McNemar’s test confirms that the performance differences between SiamCT-CA and its variants are statistically significant, with p-values < 1e-99 for most comparisons.
• The model shows strong performance on specific words like 'car' and 'hand', with SiamCT-CA+PPM achieving a notable improvement in AP for 'hand' (larger than for other words).
• Attention maps reveal that SiamCT-CA focuses attention primarily on key constraint words like 'car', while SiamCT-CA+PPM spreads attention more broadly, including on modifiers like 'black'.
• The ablation study confirms that both the attention mechanism and the Pyramid Pooling Module are critical for high performance, with the PPM improving feature aggregation across scales.
• The framework enables effective selective video compression by filtering only frames that satisfy the linguistic constraint, demonstrating practical utility in real-world applications.