[Paper Review] Active Learning for Deep Visual Tracking

Di Yuan, Xiaojun Chang+4 more|arXiv (Cornell University)|Oct 17, 2021

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes an active learning framework for deep visual tracking that selects diverse and representative video sequences for training, reducing labeling costs while maintaining competitive tracking performance. By leveraging multi-frame collaboration and nearest neighbor discrimination, the method improves sample efficiency, and with Tversky loss for bounding box regression, the tracker achieves state-of-the-art accuracy under limited training budgets.

ABSTRACT

Convolutional neural networks (CNNs) have been successfully applied to the single target tracking task in recent years. Generally, training a deep CNN model requires numerous labeled training samples, and the number and quality of these samples directly affect the representational capability of the trained model. However, this approach is restrictive in practice, because manually labeling such a large number of training samples is time-consuming and prohibitively expensive. In this paper, we propose an active learning method for deep visual tracking, which selects and annotates the unlabeled samples to train the deep CNNs model. Under the guidance of active learning, the tracker based on the trained deep CNNs model can achieve competitive tracking performance while reducing the labeling cost. More specifically, to ensure the diversity of selected samples, we propose an active learning method based on multi-frame collaboration to select those training samples that should be and need to be annotated. Meanwhile, considering the representativeness of these selected samples, we adopt a nearest neighbor discrimination method based on the average nearest neighbor distance to screen isolated samples and low-quality samples. Therefore, the training samples subset selected based on our method requires only a given budget to maintain the diversity and representativeness of the entire sample set. Furthermore, we adopt a Tversky loss to improve the bounding box estimation of our tracker, which can ensure that the tracker achieves more accurate target states. Extensive experimental results confirm that our active learning-based tracker (ALT) achieves competitive tracking accuracy and speed compared with state-of-the-art trackers on the seven most challenging evaluation benchmarks.

Motivation & Objective

• Address the high cost and time consumption of labeling large-scale training data for deep visual tracking.
• Improve the representational capability of deep CNN models under a fixed labeling budget.
• Ensure diversity and representativeness of selected training samples through temporal consistency and outlier filtering.
• Enhance bounding box estimation accuracy using a Tversky loss function to improve tracking robustness.
• Demonstrate that active learning-based sample selection enables competitive tracking performance with fewer labeled samples compared to random selection or full-data training.

Proposed method

• Propose a multi-frame collaboration-based active learning strategy to select training sequences that maximize diversity by exploiting temporal coherence in video sequences.
• Introduce a nearest neighbor discrimination method based on average nearest neighbor distance to filter out isolated or low-quality sequences, ensuring representativeness of the selected subset.
• Formulate a sample selection process that jointly optimizes diversity (via temporal consistency) and representativeness (via outlier detection) under a fixed budget.
• Integrate Tversky loss into the tracker's bounding box regression head to improve localization accuracy by penalizing false positives and false negatives asymmetrically.
• Train a deep CNN-based tracker (ALT) on the actively selected subset of video sequences, enabling high performance with minimal labeling effort.

Figure 1: A visual experimental comparison of the proposed ALT tracker with some state-of-the-art trackers. Thanks to the training samples selected by active learning and the target boundary box estimation improved by Tversky loss, our ALT tracker was able to obtain more accurate tracking results.

Experimental results

Research questions

• RQ1 Can active learning based on multi-frame collaboration improve the diversity of selected training sequences for visual tracking compared to single-frame or random selection?
• RQ2 How effective is the average nearest neighbor distance-based filtering in removing low-quality or isolated sequences to enhance representativeness?
• RQ3 Does the integration of Tversky loss lead to measurable improvements in bounding box estimation and overall tracking accuracy?
• RQ4 Can a tracker trained on a small, actively selected subset of data achieve performance comparable to state-of-the-art trackers trained on full datasets?
• RQ5 To what extent does active learning reduce labeling cost while maintaining or improving tracking performance?

Key findings

• The proposed active learning-based tracker (ALT) achieves the best EAO score on the seven benchmark datasets, outperforming state-of-the-art trackers like DiMP50 and ATOM.
• ALT achieves competitive tracking accuracy despite being trained on a significantly smaller number of labeled sequences, demonstrating the effectiveness of the active learning strategy.
• The use of Tversky loss improves bounding box estimation, leading to more accurate target localization, especially in challenging scenarios like occlusion and deformation.
• Qualitative results show that ALT maintains robust tracking performance in complex sequences (e.g., bird1, jump, soccer), outperforming SiamRPN++ and PrDiMP50 in occlusion and deformation.
• The combination of multi-frame collaboration and nearest neighbor filtering ensures that the selected training set maintains high diversity and representativeness, even under strict budget constraints.

Figure 2: An example of training sample selection under different methods: random, single frame-based active learning (SAL), multi-frame cooperation-based active learning (MAL), and neighborhood validation-based re-selection (KMAL). The bold (green) box indicates these samples selected by each of th