[Paper Review] Deeper and Wider Siamese Networks for Real-Time Visual Tracking

Zhipeng Zhang, Houwen Peng|arXiv (Cornell University)|Jan 7, 2019

Video Surveillance and Tracking MethodsComputer Science42 references110 citations

TL;DR

The paper introduces cropping-inside residual (CIR) units to remove padding-induced position bias and builds deeper/wider Siamese backbones (CIResNet family, CIResInception, CIResNeXt) for SiamFC and SiamRPN, achieving significant accuracy gains with real-time speed.

ABSTRACT

Siamese networks have drawn great attention in visual tracking because of their balanced accuracy and speed. However, the backbone networks used in Siamese trackers are relatively shallow, such as AlexNet [18], which does not fully take advantage of the capability of modern deep neural networks. In this paper, we investigate how to leverage deeper and wider convolutional neural networks to enhance tracking robustness and accuracy. We observe that direct replacement of backbones with existing powerful architectures, such as ResNet [14] and Inception [33], does not bring improvements. The main reasons are that 1)large increases in the receptive field of neurons lead to reduced feature discriminability and localization precision; and 2) the network padding for convolutions induces a positional bias in learning. To address these issues, we propose new residual modules to eliminate the negative impact of padding, and further design new architectures using these modules with controlled receptive field size and network stride. The designed architectures are lightweight and guarantee real-time tracking speed when applied to SiamFC [2] and SiamRPN [20]. Experiments show that solely due to the proposed network architectures, our SiamFC+ and SiamRPN+ obtain up to 9.8%/5.7% (AUC), 23.3%/8.8% (EAO) and 24.4%/25.0% (EAO) relative improvements over the original versions [2, 20] on the OTB-15, VOT-16 and VOT-17 datasets, respectively.

Motivation & Objective

• Analyze how backbone depth and width affect Siamese tracker performance.
• Identify factors causing performance degradation when using deeper networks.
• Propose residual modules that remove padding-induced position bias.
• Design deeper and wider CIR-based backbones under controlled receptive field and stride.
• Demonstrate real-time tracking performance with improved accuracy on standard benchmarks.

Proposed method

• Introduce cropping-inside residual (CIR) units that crop padding-affected features after the residual addition.
• Replace padding-containing backbones with CIR-based backbones (CIResNet, CIResInception, CIResNeXt) in SiamFC and SiamRPN.
• Control receptive field size and network stride to keep 60-80% RF ratio relative to exemplar size.
• Construct both deeper and wider networks using CIR units to balance localization precision and feature richness.
• Train networks with ImageNet pre-training and fine-tune in SiamFC/SiamRPN frameworks with staged unfreezing.
• Evaluate on standard tracking benchmarks (OTB, VOT) and compare against AlexNet baselines and state-of-the-art trackers.

Experimental results

Research questions

• RQ1 How do depth, width, receptive field, stride, and padding affect Siamese tracking accuracy and localization?
• RQ2 Can padding-induced position bias degrade Siamese tracking, and how can it be mitigated?
• RQ3 Do CIR-based deeper and/or wider backbones improve Siamese trackers (SiamFC, SiamRPN) in accuracy while preserving real-time speed?
• RQ4 What architectural guidelines maximize robustness and discriminability of Siamese feature embeddings?

Key findings

• Siamese trackers benefit from smaller strides (4 or 8) rather than larger ones when using deeper backbones.
• Optimal receptive field for last-layer neurons is about 60%-80% of the exemplar size, and the maximum RF should not exceed the exemplar.
• Padding in fully convolutional Siamese networks introduces position bias, leading to degraded localization near image borders.
• CIR units (and their wide variants CIR-Inception, CIR-NeXt) remove padding-affected features and improve discrimination, yielding substantial gains over AlexNet baselines.
• CIResNet-22 achieves notable gains: + up to 9.8% (AUC) on OTB-15 and +23.3% (EAO) on VOT-17 for SiamRPN/SiamFC variants, with real-time speeds (e.g., ~70–150 FPS depending on setup).
• SiamFC+ and SiamRPN+ (using CIResNet-22) outperform prior Siamese trackers on OTB-2015 and VOT-17, with SiamRPN+ reaching ~150 FPS on GTX 1080.