[Paper Review] Learning to Filter: Siamese Relation Network for Robust Tracking

Siyuan Cheng, Bineng Zhong+5 more|arXiv (Cornell University)|Apr 2, 2021

Video Surveillance and Tracking Methods49 references4 citations

TL;DR

This paper proposes a Siamese Relation Network (SRN) that enhances visual tracking robustness by introducing a Relation Detector (RD) for filtering distractors via meta-learning and a Refinement Module (RM) to align classification and regression branches. The method achieves state-of-the-art performance on five benchmarks, including VOT2018, VOT2019, OTB100, LaSOT, and UAV123, with real-time inference and improved accuracy under occlusion, fast motion, and background clutter.

ABSTRACT

Despite the great success of Siamese-based trackers, their performance under complicated scenarios is still not satisfying, especially when there are distractors. To this end, we propose a novel Siamese relation network, which introduces two efficient modules, i.e. Relation Detector (RD) and Refinement Module (RM). RD performs in a meta-learning way to obtain a learning ability to filter the distractors from the background while RM aims to effectively integrate the proposed RD into the Siamese framework to generate accurate tracking result. Moreover, to further improve the discriminability and robustness of the tracker, we introduce a contrastive training strategy that attempts not only to learn matching the same target but also to learn how to distinguish the different objects. Therefore, our tracker can achieve accurate tracking results when facing background clutters, fast motion, and occlusion. Experimental results on five popular benchmarks, including VOT2018, VOT2019, OTB100, LaSOT, and UAV123, show that the proposed method is effective and can achieve state-of-the-art results. The code will be available at https://github.com/hqucv/siamrn

Motivation & Objective

• To address the limited robustness of Siamese trackers in complex scenarios such as background clutter, occlusion, and fast motion.
• To improve discriminability by learning to distinguish targets from distractors, not just matching the same target.
• To reduce the mismatch between classification and regression branches in Siamese trackers through joint optimization.
• To enable the tracker to filter distractors without fine-tuning after initial target localization.

Proposed method

• Introduces a Relation Detector (RD) trained via few-shot, meta-learning-based contrastive training to measure relationships between target and proposal features.
• Employs a Refinement Module (RM) that fuses classification scores and RD outputs to jointly refine bounding box predictions.
• Uses a triplet-based contrastive loss during training to simultaneously learn similarity and dissimilarity between target and background samples.
• Applies hard negative mining during training to improve model robustness on challenging samples.
• Utilizes multi-level feature aggregation (from conv3, conv4, conv5) and multiple relation head types (Global, Local, Patch) for comprehensive feature comparison.
• Employs precise ROI pooling to extract proposal features for input into the RD during inference.

Figure 1: Tracking Result of our Siamese relation network with two state-of-the-art trackers in VOT2018 [ 21 ] . Benefiting from our Relation Detector and Refinement Module, when facing similar distractors, appearance change and complex background, our tracker is more robust to these challenges and

Experimental results

Research questions

• RQ1 Can a meta-learned relation detector effectively filter distractors from background in visual tracking without additional fine-tuning?
• RQ2 How does joint optimization of classification and regression via a refinement module reduce the mismatch between these branches?
• RQ3 To what extent does contrastive training improve discriminability in Siamese trackers under complex visual conditions?
• RQ4 What is the optimal combination of feature levels and relation head types for robust target-background distinction?

Key findings

• The proposed method achieves state-of-the-art performance on five standard benchmarks: VOT2018, VOT2019, OTB100, LaSOT, and UAV123.
• On OTB100, the model achieves an AUC score of 0.701 with full feature aggregation and all three relation head types (Global, Local, Patch).
• The contrastive training strategy improves AUC by 1.3% compared to naive one-way one-shot learning, demonstrating enhanced discriminability.
• Hard negative mining further boosts performance by 0.3% in AUC, indicating improved robustness on difficult samples.
• The combination of conv4 and conv5 features yields the best two-level performance, while three-level aggregation achieves the highest overall AUC.
• The tracker maintains real-time inference speed while significantly outperforming existing Siamese trackers in handling occlusion, motion blur, and background clutter.

Figure 2: The tracking pipeline of the proposed Siamese Relation Network. The proposed Relation detector (RD) and joint Refinement Module (RM) are presented. During tracking, we feed the features of proposals that are generated from the regression branch by precise ROI pooling [ 6 ] into RD to measu