[Paper Review] Ranking-Based Siamese Visual Tracking

Feng Tang, Qiang Ling|arXiv (Cornell University)|May 24, 2022

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes a ranking-based optimization (RBO) framework for Siamese visual tracking that enhances discrimination against distractors and aligns classification with localization. By introducing a classification ranking loss and an IoU-guided ranking loss, the method improves tracker robustness and accuracy across seven benchmarks, achieving state-of-the-art performance with no inference overhead.

ABSTRACT

Current Siamese-based trackers mainly formulate the visual tracking into two independent subtasks, including classification and localization. They learn the classification subnetwork by processing each sample separately and neglect the relationship among positive and negative samples. Moreover, such tracking paradigm takes only the classification confidence of proposals for the final prediction, which may yield the misalignment between classification and localization. To resolve these issues, this paper proposes a ranking-based optimization algorithm to explore the relationship among different proposals. To this end, we introduce two ranking losses, including the classification one and the IoU-guided one, as optimization constraints. The classification ranking loss can ensure that positive samples rank higher than hard negative ones, i.e., distractors, so that the trackers can select the foreground samples successfully without being fooled by the distractors. The IoU-guided ranking loss aims to align classification confidence scores with the Intersection over Union(IoU) of the corresponding localization prediction for positive samples, enabling the well-localized prediction to be represented by high classification confidence. Specifically, the proposed two ranking losses are compatible with most Siamese trackers and incur no additional computation for inference. Extensive experiments on seven tracking benchmarks, including OTB100, UAV123, TC128, VOT2016, NFS30, GOT-10k and LaSOT, demonstrate the effectiveness of the proposed ranking-based optimization algorithm. The code and raw results are available at https://github.com/sansanfree/RBO.

Motivation & Objective

• To address the challenge of background distractors in Siamese visual tracking, where hard negative samples with high confidence can mislead trackers.
• To resolve the misalignment between classification confidence and localization accuracy in existing Siamese trackers.
• To improve model generalization by explicitly modeling relationships among positive and negative proposals during training.
• To develop a plug-and-play optimization strategy compatible with various Siamese trackers without increasing inference cost.

Proposed method

• Proposes a classification ranking loss that treats classification as a ranking task, ensuring positive samples rank higher than hard negative samples (distractors).
• Introduces an IoU-guided ranking loss that aligns classification confidence scores with the Intersection over Union (IoU) of localization predictions for positive samples.
• Designs the losses to be compatible with existing Siamese trackers, requiring no architectural changes or additional inference computation.
• Employs a two-stage optimization strategy: first, the classification ranking loss suppresses distractor confidence; second, the IoU-guided loss ensures high-confidence predictions are well-localized.
• Modifies the original RankDetNet IoU-ranking loss to decouple optimization into two subtasks with explicit constraints, improving training stability and performance.
• Validates the method on both anchor-based (SiamRPN++) and anchor-free (SiamBAN) Siamese trackers, as well as Transformer-based (TransT) trackers.

Experimental results

Research questions

• RQ1 Can a ranking-based optimization strategy improve the discrimination of Siamese trackers against hard negative samples?
• RQ2 Does aligning classification confidence with IoU improve localization accuracy without increasing inference cost?
• RQ3 How effective is the proposed ranking-based optimization across diverse tracking benchmarks and tracker architectures?
• RQ4 Can the ranking-based optimization be generalized to Transformer-based visual trackers?

Key findings

• The proposed RBO method improves SiamRPN++ by 1.45% (from 60.63% to 62.08%) on the combined TC128, NFS30, and UAV123 benchmarks.
• SiamBAN-RBO achieves a 1.09% gain (from 61.15% to 62.24%) on the same benchmark set.
• The classification ranking loss effectively suppresses confidence in distractors, as shown in visualization results, outperforming two-stage hard negative mining.
• The IoU-guided ranking loss significantly improves performance, with the modified version outperforming the original RankDetNet loss due to better optimization stability.
• TransT+RBO achieves a 0.7% AUC gain on LaSOT and 0.6% on UAV123, confirming the method's effectiveness even on attention-based architectures.
• Ablation studies confirm that both ranking losses contribute independently and synergistically to performance gains, with the IoU-guided loss providing the largest improvement.