[Paper Review] POI: Multiple Object Tracking with High Performance Detection and Appearance Feature

Fengwei Yu, Wenbo Li+4 more|arXiv (Cornell University)|Oct 19, 2016

Video Surveillance and Tracking Methods10 references4 citations

TL;DR

This paper proposes POI, a high-performance multiple object tracking framework that leverages a fine-tuned Faster R-CNN detector and a deep learning-based 128D appearance feature network, achieving state-of-the-art MOTA scores of 66.1% (online) and 68.2% (offline) on the MOT16 benchmark. The method demonstrates that superior detection and learned appearance features significantly outperform traditional hand-crafted features, even with a simple online tracker architecture.

ABSTRACT

Detection and learning based appearance feature play the central role in data association based multiple object tracking (MOT), but most recent MOT works usually ignore them and only focus on the hand-crafted feature and association algorithms. In this paper, we explore the high-performance detection and deep learning based appearance feature, and show that they lead to significantly better MOT results in both online and offline setting. We make our detection and appearance feature publicly available. In the following part, we first summarize the detection and appearance feature, and then introduce our tracker named Person of Interest (POI), which has both online and offline version.

Motivation & Objective

• To improve multiple object tracking (MOT) performance by leveraging high-accuracy object detection and deep appearance features instead of relying on hand-crafted features.
• To investigate whether state-of-the-art tracking performance can be achieved with a simple online tracker when combined with high-quality detection and appearance features.
• To demonstrate that the performance gap between online and offline trackers diminishes when using high-quality detection and appearance features.
• To release a high-quality detection model and deep appearance feature network for public use to benefit future MOT research.

Proposed method

• A Faster R-CNN detector is fine-tuned on a combination of ImageNet, ETHZ, Caltech, and a self-collected surveillance dataset using multi-scale training and skip pooling to improve detection accuracy.
• A GoogLeNet-inspired CNN is trained to extract 128D appearance features from person crops, using joint softmax and triplet loss to enhance discriminative power and intra-class compactness.
• The online tracker uses Kalman filtering for motion prediction and the Kuhn-Munkres algorithm for data association, combining appearance, motion, and shape affinities into a final affinity score.
• Affinity is computed as the product of cosine distance (appearance), exponential decay of normalized position error (motion), and normalized size difference (shape), with learned weights.
• A two-stage tracking quality thresholding mechanism separates high- and low-quality tracklets to improve association accuracy and reduce false positives.
• A scale-aware strategy with thresholds τs and τr is introduced to handle large targets by suppressing unreliable motion and smoothness affinities for oversized detections.

Experimental results

Research questions

• RQ1 Can high-performance detection and deep appearance features significantly improve MOT performance in both online and offline settings?
• RQ2 Does the use of learned appearance features eliminate the need for complex offline tracking pipelines, reducing the performance gap between online and offline trackers?
• RQ3 How do detection quality and appearance feature quality individually and jointly affect key MOT metrics like MOTA, IDS, and FP?
• RQ4 To what extent do scale variations in target size affect tracking performance, and can this be mitigated with adaptive affinity weighting?
• RQ5 Can a simple online tracker achieve SOTA performance when equipped with high-quality detection and deep appearance features?

Key findings

• The proposed detector reduces the sum of false positives and false negatives (FP+FN) from 91,192 (DPMv5) to 46,587, significantly improving detection quality.
• The online tracker achieves a MOTA of 66.1% on the MOT16 test set, outperforming the state-of-the-art method KFILDAwSDP (55.2%) and MCMOT-HDM (62.4%).
• The offline tracker achieves a MOTA of 68.2%, setting a new SOTA on the MOT16 benchmark, with the lowest FN (45,605) among all methods.
• The combination of the proposed detector and deep appearance feature reduces IDS from 784 (DPMv5 + our feature) to 716 and FM from 2,697 to 1,973 on the training set.
• The online tracker with the proposed detector and feature achieves a MOTA of 65.2% on the training set, significantly outperforming the baseline (29.4%) and showing strong generalization.
• The ablation study confirms that both detection and appearance feature quality are critical: using only the proposed detector or only the feature yields suboptimal results, but their combination achieves SOTA performance.