[Paper Review] UA-DETRAC: A New Benchmark and Protocol for Multi-Object Detection and Tracking

Longyin Wen, Dawei Du+7 more|arXiv (Cornell University)|Nov 13, 2015

Video Surveillance and Tracking Methods74 references142 citations

TL;DR

This paper introduces UA-DETRAC, a large-scale benchmark with 100 real-world traffic videos and 140,000+ annotated frames for multi-object detection and tracking. It proposes a new evaluation protocol that decouples detection and tracking, revealing the critical impact of detection quality on overall MOT performance, and introduces improved metrics like PR-MOTA to better assess system robustness across diverse scenarios.

ABSTRACT

In recent years, numerous effective multi-object tracking (MOT) methods are developed because of the wide range of applications. Existing performance evaluations of MOT methods usually separate the object tracking step from the object detection step by using the same fixed object detection results for comparisons. In this work, we perform a comprehensive quantitative study on the effects of object detection accuracy to the overall MOT performance, using the new large-scale University at Albany DETection and tRACking (UA-DETRAC) benchmark dataset. The UA-DETRAC benchmark dataset consists of 100 challenging video sequences captured from real-world traffic scenes (over 140,000 frames with rich annotations, including occlusion, weather, vehicle category, truncation, and vehicle bounding boxes) for object detection, object tracking and MOT system. We evaluate complete MOT systems constructed from combinations of state-of-the-art object detection and object tracking methods. Our analysis shows the complex effects of object detection accuracy on MOT system performance. Based on these observations, we propose new evaluation tools and metrics for MOT systems that consider both object detection and object tracking for comprehensive analysis.

Motivation & Objective

• Address the lack of comprehensive evaluation protocols that account for detection accuracy in multi-object tracking (MOT) systems.
• Provide a large-scale, real-world dataset with rich annotations (e.g., occlusion, illumination, vehicle types) to support robust evaluation of MOT methods.
• Investigate the complex interplay between object detection quality and overall MOT system performance, challenging the assumption of fixed detection inputs.
• Develop new evaluation metrics that consider detection performance as a variable, enabling fairer and more informative benchmarking across diverse application scenarios.
• Enable future research on joint detection and tracking, real-time efficiency, and data-driven performance gains by establishing a standardized, extensible benchmark framework.

Proposed method

• Collect and annotate 100 real-world traffic videos (over 140,000 frames) with bounding boxes, occlusion levels, truncation ratios, illumination conditions, and vehicle types.
• Construct complete MOT systems by combining state-of-the-art detectors (DPM, ACF, R-CNN, CompACT, Faster R-CNN) with ten tracking algorithms (GOG, CEM, DCT, IHTLS, H2T, CMOT, TBD).
• Evaluate MOT performance across varying detection confidence thresholds to analyze the sensitivity of tracking performance to detection quality.
• Introduce a new evaluation protocol that decouples detection and tracking, allowing systematic analysis of detection’s impact on tracking metrics like MOTA and MOTP.
• Propose enhanced metrics such as PR-MOTA and PR-MOTP that incorporate precision and recall of detection inputs, enabling application-specific performance assessment.
• Use frame-level speed measurements and computational platforms to evaluate real-time feasibility of detection and tracking pipelines.

Experimental results

Research questions

• RQ1 How does the accuracy of object detection influence the overall performance of multi-object tracking systems?
• RQ2 What are the limitations of existing evaluation protocols that treat detection as a fixed, predefined component in MOT benchmarks?
• RQ3 Can a new evaluation protocol be designed to systematically analyze the impact of detection quality on tracking performance?
• RQ4 How do different detection algorithms affect the MOTA and MOTP scores of downstream tracking systems across diverse real-world conditions?
• RQ5 What metrics are most informative for evaluating MOT systems in application-specific contexts such as surveillance (minimize identity switches) versus autonomous driving (minimize false negatives)?

Key findings

• Detection quality has a significant and non-linear impact on MOT system performance, with MOTA scores varying substantially across different detection confidence thresholds.
• The proposed PR-MOTA metric reveals that tracking performance is highly sensitive to detection precision and recall, especially under challenging conditions like occlusion and low illumination.
• Faster R-CNN and CompACT detectors consistently outperform DPM and ACF in terms of detection quality, leading to higher MOTA scores when paired with robust trackers.
• The GOG tracker achieves the highest frame rate (404.61 FPS) on the UA-DETRAC test set when using Faster R-CNN detections, indicating strong real-time potential.
• The IHTLS tracker shows the highest sensitivity to detection quality, with MOTA dropping significantly at low detection recall, highlighting the need for robust detection in complex tracking scenarios.
• The analysis confirms that joint optimization of detection and tracking is essential, as tracking performance is not independent of detection quality, and vice versa.