[Paper Review] VITAL: VIsual Tracking via Adversarial Learning

Yibing Song, Chao Ma+7 more|arXiv (Cornell University)|Apr 12, 2018

Video Surveillance and Tracking MethodsComputer Science47 references58 citations

TL;DR

VITAL augments positive samples with feature-space adversarial masks and uses a cost-sensitive loss to address class imbalance, improving robustness in tracking-by-detection against state-of-the-art methods.

ABSTRACT

The tracking-by-detection framework consists of two stages, i.e., drawing samples around the target object in the first stage and classifying each sample as the target object or as background in the second stage. The performance of existing trackers using deep classification networks is limited by two aspects. First, the positive samples in each frame are highly spatially overlapped, and they fail to capture rich appearance variations. Second, there exists extreme class imbalance between positive and negative samples. This paper presents the VITAL algorithm to address these two problems via adversarial learning. To augment positive samples, we use a generative network to randomly generate masks, which are applied to adaptively dropout input features to capture a variety of appearance changes. With the use of adversarial learning, our network identifies the mask that maintains the most robust features of the target objects over a long temporal span. In addition, to handle the issue of class imbalance, we propose a high-order cost sensitive loss to decrease the effect of easy negative samples to facilitate training the classification network. Extensive experiments on benchmark datasets demonstrate that the proposed tracker performs favorably against state-of-the-art approaches.

Motivation & Objective

• Address overfitting to frame-specific discriminative features in tracking-by-detection
• Augment positive samples in the feature space to capture temporal appearance variations
• Mitigate class imbalance by introducing a high-order cost-sensitive loss
• Leverage adversarial learning to identify robust features over long temporal spans
• Demonstrate favorable performance against state-of-the-art trackers on standard benchmarks

Proposed method

• Insert a generative network G between the last CNN conv layer and the first FC layer to produce a weight mask for feature dropout
• Train D (the classifier) with masked features to learn temporally robust representations
• Use adversarial learning where G iteratively identifies masks that maximize D’s loss, guiding D away from frame-specific discriminative features
• Apply a high-order cost-sensitive loss (based on a focal-like modulation) to down-weight easy negatives and emphasize hard negatives
• Train G and D in an alternating fashion during offline pretraining and online updates, but remove G at test time
• Pretrain with labeled samples and online fine-tune using diversified positive samples and mined hard negatives

Experimental results

Research questions

• RQ1 Can adversarially generated feature masks achieve more temporally robust representations for tracking-by-detection?
• RQ2 Does a high-order cost-sensitive loss improve discrimination under extreme class imbalance in tracking datasets?
• RQ3 How does VITAL compare to state-of-the-art trackers on standard benchmarks in precision and overlap metrics?
• RQ4 Is temporal robustness achieved by focusing on features that persist over time rather than frame-specific discriminative cues?

Key findings

• VITAL achieves favorable results against state-of-the-art trackers on standard benchmarks like OTB-2013, OTB-2015, and VOT-2016
• Adversarially learned masks reduce reliance on frame-specific discriminative features and promote temporally robust representations
• The proposed cost-sensitive loss helps mining hard negatives and mitigating easy-negative dominance during training
• Ablation studies show random masking degrades performance, while adversarially learned masking improves robustness and accuracy
• VITAL shows improved performance under challenging conditions such as occlusion, deformation, and viewpoint changes compared to several baselines