[Paper Review] MOT20: A benchmark for multi object tracking in crowded scenes

Patrick Dendorfer, Hamid Rezatofighi+7 more|arXiv (Cornell University)|Mar 19, 2020

Video Surveillance and Tracking Methods16 references510 citations

TL;DR

MOT20 introduces eight highly crowded pedestrian sequences to extend MOTChallenge, providing standardized annotations, public detections, and evaluation protocols to stress-test trackers in extremely crowded scenes.

ABSTRACT

Standardized benchmarks are crucial for the majority of computer vision applications. Although leaderboards and ranking tables should not be over-claimed, benchmarks often provide the most objective measure of performance and are therefore important guides for research. The benchmark for Multiple Object Tracking, MOTChallenge, was launched with the goal to establish a standardized evaluation of multiple object tracking methods. The challenge focuses on multiple people tracking, since pedestrians are well studied in the tracking community, and precise tracking and detection has high practical relevance. Since the first release, MOT15, MOT16, and MOT17 have tremendously contributed to the community by introducing a clean dataset and precise framework to benchmark multi-object trackers. In this paper, we present our MOT20benchmark, consisting of 8 new sequences depicting very crowded challenging scenes. The benchmark was presented first at the 4thBMTT MOT Challenge Workshop at the Computer Vision and Pattern Recognition Conference (CVPR) 2019, and gives to chance to evaluate state-of-the-art methods for multiple object tracking when handling extremely crowded scenarios.

Motivation & Objective

• Provide a challenging, standardized benchmark for multi-object tracking in crowded scenes.
• Extend previous MOTChallenge releases with higher pedestrian density to test generalization and robustness.
• Offer careful annotations, public detections, and a consistent evaluation framework to enable fair tracker comparison.

Proposed method

• Define target classes and annotation rules focusing on moving pedestrians while excluding distractors from evaluation.
• Curate eight high-density sequences (up to 246 pedestrians per frame) across indoor/outdoor, day/night conditions.
• Provide training/testing splits with ground-truth annotations for training and public detections for tracking evaluation.
• Use a standardized data format (CSV) for detections and annotations and deliver a ZIP per sequence for submission.
• Adopt CLEAR metrics and track-quality measures (MOTA, MOTP, MT/PT/ML, ID switches, fragmentations) for comprehensive evaluation.
• Present a Faster R-CNN (ResNet101) detector trained on MOT20 training data as a public baseline.

Experimental results

Research questions

• RQ1 How do state-of-the-art trackers perform on extremely crowded crowd scenes?
• RQ2 Do detectors and trackers generalize across scenes and conditions unseen during training?
• RQ3 How do traditional MOT metrics (MOTA, MOTP) relate to track quality measures (MT/PT/ML, ID switches) in dense crowds?
• RQ4 What is the impact of using public detections versus private detections on tracker evaluation?
• RQ5 How robust are tracking methods to occlusions and high-density scenarios?

Key findings

• The MOT20 dataset contains 8 sequences from 3 scenes with densities up to 246 pedestrians per frame.
• Public Faster R-CNN detections (trained on MOT20 training data) were provided as baselines for tracking evaluation.
• Training sequences yield a combined 1,134,614 boxes; testing sequences yield 517,426 boxes across frames.
• Reported MOT scores show variability across sequences, e.g., MOT20-01 to MOT20-05 have APs between 0.63–0.82 and MT/ML variations, reflecting crowding challenges.
• Detections per sequence vary widely, from ~12k to ~381k detections, with min/max heights indicating diverse scales.
• Across the eight sequences, the detector-based baseline achieved varying APs (0.38–0.82) and MOTA/MOTP values, illustrating the challenge of crowded scenes.