[Paper Review] Learning to Fuse Asymmetric Feature Maps in Siamese Trackers

Wencheng Han, Xingping Dong+3 more|arXiv (Cornell University)|Dec 4, 2020

Video Surveillance and Tracking Methods51 references4 citations

TL;DR

This paper proposes a learnable asymmetric convolution module (ACM) that improves feature fusion in Siamese trackers by learning to correlate asymmetric feature maps from the target template and search region. Unlike handcrafted DW-XCorr, ACM enables end-to-end training with large-scale data, integrates bounding-box size priors, and achieves state-of-the-art performance, with a 5.8% AUC gain on LaSOT and consistent improvements across six benchmarks.

ABSTRACT

Recently, Siamese-based trackers have achieved promising performance in visual tracking. Most recent Siamese-based trackers typically employ a depth-wise cross-correlation (DW-XCorr) to obtain multi-channel correlation information from the two feature maps (target and search region). However, DW-XCorr has several limitations within Siamese-based tracking: it can easily be fooled by distractors, has fewer activated channels, and provides weak discrimination of object boundaries. Further, DW-XCorr is a handcrafted parameter-free module and cannot fully benefit from offline learning on large-scale data. We propose a learnable module, called the asymmetric convolution (ACM), which learns to better capture the semantic correlation information in offline training on large-scale data. Different from DW-XCorr and its predecessor(XCorr), which regard a single feature map as the convolution kernel, our ACM decomposes the convolution operation on a concatenated feature map into two mathematically equivalent operations, thereby avoiding the need for the feature maps to be of the same size (width and height)during concatenation. Our ACM can incorporate useful prior information, such as bounding-box size, with standard visual features. Furthermore, ACM can easily be integrated into existing Siamese trackers based on DW-XCorror XCorr. To demonstrate its generalization ability, we integrate ACM into three representative trackers: SiamFC, SiamRPN++, and SiamBAN. Our experiments reveal the benefits of the proposed ACM, which outperforms existing methods on six tracking benchmarks. On the LaSOT test set, our ACM-based tracker obtains a significant improvement of 5.8% in terms of success (AUC), over the baseline.

Motivation & Objective

• To address the limitations of handcrafted DW-XCorr in Siamese trackers, such as vulnerability to distractors and weak boundary discrimination.
• To enable end-to-end learning of correlation modules by replacing parameter-free DW-XCorr with a learnable alternative.
• To allow fusion of asymmetric feature maps (different spatial sizes) without requiring size alignment.
• To integrate non-visual priors, such as bounding-box size, into the correlation learning process.
• To improve tracking robustness and accuracy across diverse benchmarks without sacrificing speed.

Proposed method

• Proposes an asymmetric convolution (ACM) that decomposes cross-correlation into two mathematically equivalent operations, enabling fusion of feature maps with different spatial dimensions.
• Designs a learnable module that replaces DW-XCorr in Siamese trackers, allowing offline pre-training on large-scale data to improve feature representation.
• Introduces a mechanism to concatenate bounding-box size as additional prior information with visual features in the correlation layer.
• Employs a two-branch architecture where the template and search region features are processed separately before correlation via ACM.
• Uses a learnable kernel that operates on a concatenated feature map, avoiding the need for feature maps to be of equal size during concatenation.
• Integrates ACM into three SOTA Siamese trackers: SiamFC, SiamRPN++, and SiamBAN, with minimal architectural changes.

Figure 1: Comparison between DW-XCorr and ACM in terms of being fooled by distractors (first row), information distribution across channels (second row) and background suppression to better discriminative target boundaries (third row). DW-XCorr produces similar responses for distractors and the targ

Experimental results

Research questions

• RQ1 Can a learnable correlation module outperform handcrafted DW-XCorr in Siamese trackers by leveraging large-scale pre-training?
• RQ2 How does incorporating bounding-box size as a prior improve tracking performance and robustness?
• RQ3 Can a correlation module effectively fuse asymmetric feature maps without requiring spatial size alignment?
• RQ4 Does the proposed ACM reduce false positives from distractors and improve boundary localization?
• RQ5 To what extent does ACM improve performance across diverse benchmarks like LaSOT, VOT, and OTB-100?

Key findings

• On the LaSOT benchmark, the ACM-based tracker achieves a 5.8% absolute improvement in success AUC over the baseline, demonstrating strong generalization.
• On VOT2016, the SiamBAN-ACM tracker achieves a 4.4% absolute gain in EAO over the baseline SiamBAN, with a 20% lower failure rate.
• On VOT2019, the ACM-based trackers show consistent performance gains, with improved accuracy and robustness compared to baselines.
• On UAV123, the SiamBAN-ACM achieves an AUC score of 64.8, outperforming SiamCAR (61.4) and SiamBAN (63.1).
• The ablation study confirms that both the AC module and the integration of bounding-box size contribute to performance gains, with the full ACM outperforming variants without priors.
• The ACM-based trackers maintain high inference speed, with SiamFC-ACM running at 172 FPS on a GTX1080Ti, showing minimal speed cost.

Figure 2: Comparison of (c) AC with (a) DW-XCorr and (b) a naive strategy to fuse different-sized feature maps. (a): DW-XCorr uses a $C$ channel feature map extracted from the template as kernel and convolves instance feature maps in a depth-wise manner to generate a $C$ channel correlation feature