[Paper Review] Simple Online and Realtime Tracking with a Deep Association Metric

Nicolai Wojke, Alex Bewley+1 more|arXiv (Cornell University)|Mar 21, 2017

Video Surveillance and Tracking Methods17 references119 citations

TL;DR

Extends SORT with a deep appearance metric to improve online multi-object tracking; uses a pre-trained CNN for appearance descriptors and a matching cascade to reduce identity switches by about 45% while maintaining real-time performance.

ABSTRACT

Simple Online and Realtime Tracking (SORT) is a pragmatic approach to multiple object tracking with a focus on simple, effective algorithms. In this paper, we integrate appearance information to improve the performance of SORT. Due to this extension we are able to track objects through longer periods of occlusions, effectively reducing the number of identity switches. In spirit of the original framework we place much of the computational complexity into an offline pre-training stage where we learn a deep association metric on a large-scale person re-identification dataset. During online application, we establish measurement-to-track associations using nearest neighbor queries in visual appearance space. Experimental evaluation shows that our extensions reduce the number of identity switches by 45%, achieving overall competitive performance at high frame rates.

Motivation & Objective

• Improve online multi-object tracking robustness to occlusions by incorporating appearance information.
• Maintain simplicity and real-time performance comparable to SORT.
• Leverage offline-trained deep appearance features to guide online data association.
• Evaluate on MOT16 to quantify reductions in identity switches and overall tracking metrics.

Proposed method

• Maintain Kalman filter-based single-hypothesis tracking with frame-by-frame data association.
• Integrate two data-association metrics: (i) Mahalanobis distance for motion-based gating; (ii) cosine distance in appearance space using deep CNN descriptors.
• Compute a combined cost c_{i,j} = lambda * d^(1)(i,j) + (1 - lambda) * d^(2)(i,j) and gate associations with both metrics.
• Use a matching cascade that prioritizes recently observed tracks by age to solve a sequence of subproblems and reduce fragmentation.
• Employ a deep CNN to generate 128-dimensional L2-normalized appearance descriptors from cropped bounding boxes; features trained on large-scale person re-id dataset; online nearest-neighbor matching in appearance space.
• Provide pre-trained CNN model and code for feature extraction to enable online tracking.

Experimental results

Research questions

• RQ1 Can appearance information learned offline improve online tracking performance of SORT without sacrificing real-time speed?
• RQ2 How should motion and appearance cues be fused to maximize identity preservation during occlusions and detections misses?
• RQ3 What is the impact of appearance-based data association on identity switches and track fragmentation in online tracking?”
• RQ4 Is the extended SORT competitive with state-of-the-art online trackers on MOT16 using standard detections?

Key findings

• Identity switches reduced from 1423 (SORT) to 781 (Deep SORT), a ~45% decrease.
• Deep SORT achieves competitive MOT scores while reducing ID switches and maintaining online processing at ~20 Hz.
• In MOT16 results, Deep SORT (online) reports MOTA 61.4, MOTP 79.1, MT 32.8%, ML 18.2%, ID 781, FM 2008, FP 12852, FN 56668, Runtime 40 Hz.
• The method maintains identities through longer occlusions by leveraging appearance information.
• The approach is fast enough for real-time use when GPU resources are available, with about half the time spent on feature extraction.