[Paper Review] Context-aware Visual Tracking with Joint Meta-updating

Qiuhong Shen, Xin Li+2 more|arXiv (Cornell University)|Apr 4, 2022

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes a context-aware visual tracking framework that jointly meta-updates both the localization and box-estimation branches of a Siamese tracker in representation space, leveraging global and local feature complementarity via a context-aggregation module and affinity vectors. The method achieves an EAO score of 0.514 on VOT2018 at 40 FPS, significantly outperforming baseline trackers while mitigating overfitting through gradient-based meta-learning.

ABSTRACT

Visual object tracking acts as a pivotal component in various emerging video applications. Despite the numerous developments in visual tracking, existing deep trackers are still likely to fail when tracking against objects with dramatic variation. These deep trackers usually do not perform online update or update single sub-branch of the tracking model, for which they cannot adapt to the appearance variation of objects. Efficient updating methods are therefore crucial for tracking while previous meta-updater optimizes trackers directly over parameter space, which is prone to over-fit even collapse on longer sequences. To address these issues, we propose a context-aware tracking model to optimize the tracker over the representation space, which jointly meta-update both branches by exploiting information along the whole sequence, such that it can avoid the over-fitting problem. First, we note that the embedded features of the localization branch and the box-estimation branch, focusing on the local and global information of the target, are effective complements to each other. Based on this insight, we devise a context-aggregation module to fuse information in historical frames, followed by a context-aware module to learn affinity vectors for both branches of the tracker. Besides, we develop a dedicated meta-learning scheme, on account of fast and stable updating with limited training samples. The proposed tracking method achieves an EAO score of 0.514 on VOT2018 with the speed of 40FPS, demonstrating its capability of improving the accuracy and robustness of the underlying tracker with little speed drop.

Motivation & Objective

• To address the limitations of existing deep trackers that fail under large appearance variations due to single-branch online updates or fixed offline templates.
• To overcome overfitting in parameter-space meta-updating by shifting optimization to the representation space using sequence-wide context.
• To enhance tracking performance by jointly updating both localization and box-estimation branches using complementary local and global features.
• To develop a stable, fast online adaptation mechanism using a dedicated meta-initialization scheme with limited training samples.
• To improve robustness and accuracy in long-term tracking sequences without significant speed degradation.

Proposed method

• Introduces a context-aggregation module that fuses historical frame features to enrich representation for both tracker branches.
• Designs a context-aware module that learns task-specific affinity vectors to guide joint adaptation of localization and regression branches.
• Employs a gradient-based meta-learning scheme to initialize and update the joint updater, ensuring fast and stable adaptation with few samples.
• Optimizes the tracker in representation space rather than parameter space, reducing overfitting risk during long sequences.
• Uses Siamese network architecture with separate branches for classification (local features) and bounding box regression (global features), enhanced by cross-branch attention via affinity vectors.
• Applies meta-initialization to the joint updater to stabilize early adaptation and improve generalization on unseen tracking sequences.

Experimental results

Research questions

• RQ1 Can joint meta-updating of localization and box-estimation branches improve tracking robustness under large appearance variations?
• RQ2 Does optimizing in representation space instead of parameter space reduce overfitting during long-term tracking?
• RQ3 Can complementary local and global features be effectively fused to enhance tracker adaptation?
• RQ4 How does a meta-initialization scheme improve the stability and speed of online adaptation in low-data regimes?
• RQ5 To what extent does the proposed method outperform state-of-the-art trackers on diverse benchmarks?

Key findings

• The proposed method achieves an EAO score of 0.514 on VOT2018, representing a 16.8% relative improvement over the DiMP baseline.
• On LaSOT, the method achieves an AUC score of 0.573, outperforming all compared state-of-the-art trackers across success, precision, and normalized precision metrics.
• The ablation study confirms that joint updating of both branches yields a 0.506 EAO on VOT2018, significantly outperforming single-branch updates (0.455–0.473).
• The inclusion of meta-initialization increases the AUC score to 0.697 on OTB100 and EAO to 0.514 on VOT2018, demonstrating its stabilizing effect.
• The model maintains high efficiency with 40 FPS inference speed, showing minimal performance drop despite enhanced adaptation capability.
• The context-aware module with affinity vectors improves both localization accuracy and box estimation, especially under high appearance variation.