[Paper Review] Tracking the Trackers: An Analysis of the State of the Art in Multiple Object Tracking

Laura Leal-Taixé, Anton Milan+4 more|arXiv (Cornell University)|Apr 10, 2017

Video Surveillance and Tracking Methods50 references94 citations

TL;DR

The paper introduces MOT15 and MOT16, standardized benchmarks for multi-object tracking, analyzes 32 trackers on MOT15 and 16 on MOT16, and investigates evaluation metrics and predictors of performance.

ABSTRACT

Standardized benchmarks are crucial for the majority of computer vision applications. Although leaderboards and ranking tables should not be over-claimed, benchmarks often provide the most objective measure of performance and are therefore important guides for research. We present a benchmark for Multiple Object Tracking launched in the late 2014, with the goal of creating a framework for the standardized evaluation of multiple object tracking methods. This paper collects the two releases of the benchmark made so far, and provides an in-depth analysis of almost 50 state-of-the-art trackers that were tested on over 11000 frames. We show the current trends and weaknesses of multiple people tracking methods, and provide pointers of what researchers should be focusing on to push the field forward.

Motivation & Objective

• Introduce a standardized benchmark for fair evaluation of multi-target tracking methods (MOT15 and MOT16).
• Analyze the performance of a large set of state-of-the-art trackers on MOT15 and MOT16.
• Examine the evaluation metrics with human evaluators to assess alignment with perceived quality.
• Identify main weaknesses of current trackers and provide guidance for future research.

Proposed method

• Assemble MOT15 (22 sequences, 101k+ bounding boxes) and MOT16 (14 sequences, 292k+ boxes) with a strict annotation protocol across 12 classes.
• Provide detections (ACF for MOT15, DPM/off-the-shelf for MOT16) and require trackers to operate on these detections.
• Centralized evaluation server to ensure fair, consistent metric computation and prevent overfitting to test sequences.
• Evaluate 32 trackers on MOT15 and 16 on MOT16 (pre-2017 publications).
• Perform error analysis by comparing tracker FP/FN against detector performance to understand error sources.
• Experiment with predicting which tracker performs best on a given sequence or snippet using features and a simple SVM predictor.

Experimental results

Research questions

• RQ1 What are the current strengths and weaknesses of state-of-the-art multi-object trackers on MOT15 and MOT16?
• RQ2 How do different affinity models and appearance cues impact tracking performance?
• RQ3 To what extent do evaluation metrics reflect human visual judgments of tracking quality?
• RQ4 Can a per-sequence or per-snippet predictor select the best tracker to form a ‘super tracker’?
• RQ5 What are the limits and potential improvements suggested by standardized benchmarks?

Key findings

• Top trackers (e.g., NOMT, JMC, MDPNN16, oICF, MHT_DAM, LINF1) achieve MOTA above 40% and Mostly Tracked >10%.
• Performance is driven largely by affinity/appearance models, with deeply learned approaches giving strong results.
• Most methods reduce false positives from the detector but often fail to reduce false negatives, leading to FN-dominated MOTA challenges.
• There is high cross-video correlation: methods tend to perform similarly across sequences or fragments.
• MOTA remains the most representative metric aligned with human visual assessment, though MT and Recall also correlate with perceived quality.
• Attempting to create a ‘super tracker’ by segment-wise tracker selection yields modest gains (5.5 points on MOT15, 2.9 on MOT16) with oracle guidance; real-prediction gains are smaller.
• Human evaluators largely agree with MOTA as the best single metric for overall tracker quality.