[Paper Review] Multi-Object Tracking with Multiple Cues and Switcher-Aware Classification

Weitao Feng, Zhihao Hu+3 more|arXiv (Cornell University)|Jan 18, 2019

Video Surveillance and Tracking MethodsComputer Science39 references90 citations

TL;DR

Proposes a unified MOT framework that fuses short-term cues from a Single Object Tracker (SOT) and long-term cues from a ReID network with a switcher-aware classifier to improve data association and reduce identity switches; achieves state-of-the-art results on MOT16/17 benchmarks.

ABSTRACT

In this paper, we propose a unified Multi-Object Tracking (MOT) framework learning to make full use of long term and short term cues for handling complex cases in MOT scenes. Besides, for better association, we propose switcher-aware classification (SAC), which takes the potential identity-switch causer (switcher) into consideration. Specifically, the proposed framework includes a Single Object Tracking (SOT) sub-net to capture short term cues, a re-identification (ReID) sub-net to extract long term cues and a switcher-aware classifier to make matching decisions using extracted features from the main target and the switcher. Short term cues help to find false negatives, while long term cues avoid critical mistakes when occlusion happens, and the SAC learns to combine multiple cues in an effective way and improves robustness. The method is evaluated on the challenging MOT benchmarks and achieves the state-of-the-art results.

Motivation & Objective

• Leverage both short-term (SOT-based) and long-term (ReID-based) cues for robust MOT.
• Develop a switcher-aware classifier to reduce identity switches during data association.
• Integrate cue fusion and adaptive selection to handle occlusions and false negatives.
• Evaluate the approach on MOT16 and MOT17 to demonstrate state-of-the-art performance.

Proposed method

• Use Siamese-RPN as the SOT to capture short-term cues and generate a SOT box and score.
• Employ a ReID sub-net (GoogLeNet Inception-v4 based) to extract long-term appearance features from tracklet history.
• Propose a switcher-aware classifier (SAC) implemented with boosting decision trees to fuse short-term and long-term cues, including the switcher’s features, for matching detections to targets.
• Construct a bipartite graph of targets and detections with matching scores from SAC and solve via minimum-cost network flow for online data association.
• Select K past tracklet images with a quality-aware mechanism to form long-term cues, and compute long-term similarity using normalized ReID features.
• Incorporate a quality-based tracklet history and a detection score refinement step to mitigate noisy detections.

Experimental results

Research questions

• RQ1 How can short-term and long-term cues be effectively combined to improve data association in MOT?
• RQ2 Does incorporating switcher information into the classifier reduce identity switches and improve IDF1?
• RQ3 What is the impact of using SOT and ReID together versus individually on MOTA and ID metrics?
• RQ4 Can a learning-based switching-aware approach outperform traditional pairwise affinity methods in MOT?

Key findings

• The framework achieves competitive or state-of-the-art MOTA and IDF1 on MOT16/MOT17 benchmarks. (Exact numbers are reported in the results tables.)
• Short-term cues provide notable gains in MOTA, while long-term cues significantly boost IDF1, with their combination yielding the best overall performance.
• Incorporating the Switcher-Aware Classifier (SAC) substantially reduces identity switches (IDS) and improves IDF1, with modest impact on MOTA.
• Using a boosting decision tree classifier for SAC provides better performance for small feature dimensions compared to other classifiers (NN, SVM).
• A quality-aware tracklet history and detection score refinement contribute to robustness against noisy detections.