[Paper Review] Describe and Attend to Track: Learning Natural Language guided Structural Representation and Visual Attention for Object Tracking

Xiao Wang, Chenglong Li+4 more|arXiv (Cornell University)|Nov 25, 2018

Video Surveillance and Tracking Methods39 references26 citations

TL;DR

This paper proposes DAT (Describe and Attend to Track), a novel visual tracking framework that integrates natural language descriptions and graph convolutional networks (GCN) to enhance feature representation and visual attention. By modeling relationships among training samples via GCN and using language-guided triplet loss, the method improves robustness to occlusion and appearance variation, achieving state-of-the-art performance on five benchmark datasets with a 67.1% success rate on OTB2013 using 3 GCN layers.

ABSTRACT

The tracking-by-detection framework requires a set of positive and negative training samples to learn robust tracking models for precise localization of target objects. However, existing tracking models mostly treat different samples independently while ignores the relationship information among them. In this paper, we propose a novel structure-aware deep neural network to overcome such limitations. In particular, we construct a graph to represent the pairwise relationships among training samples, and additionally take the natural language as the supervised information to learn both feature representations and classifiers robustly. To refine the states of the target and re-track the target when it is back to view from heavy occlusion and out of view, we elaborately design a novel subnetwork to learn the target-driven visual attentions from the guidance of both visual and natural language cues. Extensive experiments on five tracking benchmark datasets validated the effectiveness of our proposed method.

Motivation & Objective

• To address the limitations of existing tracking-by-detection methods that treat training samples independently, ignoring inter-sample relationships.
• To improve robustness to heavy occlusion, large deformation, and out-of-view scenarios in visual tracking.
• To leverage natural language descriptions as high-level semantic supervision to guide structural feature learning and attention generation.
• To design a target-driven global attention mechanism that enables effective re-detection after tracking failures.
• To integrate both local and global proposal generation strategies for improved tracking accuracy and robustness.

Proposed method

• Construct a graph where each training sample is a node, and use graph convolutional networks (GCN) to propagate and refine pairwise relationship features across samples.
• Use a triplet loss function with natural language embeddings to guide the learning of structural representations, enhancing discriminative ability.
• Design a novel subnetwork, GPGNet, to generate target-specific visual attention maps using both visual patches and natural language specifications.
• Concatenate features from the global attention regions with local proposals and feed them into a binary classifier for final tracking decision.
• Employ an end-to-end training scheme that jointly optimizes the GCN-based structural representation and the attention-guided proposal generation.
• Use a lightweight convolutional encoder for efficient feature extraction from frames, language, and target patches, followed by feature concatenation and upsampling to generate attention maps.

Experimental results

Research questions

• RQ1 Can modeling inter-sample relationships through a graph structure improve the discriminative power of visual tracking features?
• RQ2 Can natural language supervision enhance the robustness of tracking models under challenging conditions such as occlusion and appearance variation?
• RQ3 Can target-driven visual attention based on both visual and linguistic cues improve re-detection after target loss?
• RQ4 How does the integration of global and local search strategies affect tracking performance on long-term tracking benchmarks?
• RQ5 What is the optimal number of GCN layers for balancing accuracy and training efficiency in visual tracking?

Key findings

• The proposed DAT tracker achieves a 67.1% success rate on the OTB2013 benchmark when using 5 GCN layers, outperforming the baseline pyMDNet (65.4%) and other state-of-the-art methods.
• The model achieves 91.8% precision and 65.2% success rate on a challenging sub-dataset of 46 OTB100 sequences with similar-looking distractors, significantly outperforming pyMDNet (86.5% precision, 64.2% success).
• Using 3 GCN layers yields the best trade-off between performance and training time, with a 0.663 success rate on OTB2013, slightly better than 2 layers (0.654) and comparable to 5 layers (0.671).
• The integration of language-guided triplet loss and GCN-based structural modeling leads to a significant performance gain, especially in handling hard positive and negative samples.
• The target-driven global attention mechanism effectively recovers targets after heavy occlusion and out-of-view events, as demonstrated by improved performance on long-term tracking sequences.
• The proposed GPGNet subnetwork successfully generates video-specific attention maps that focus on the target object, unlike generic saliency maps, and enables effective global proposal generation.