[Paper Review] ByteTrack: Multi-Object Tracking by Associating Every Detection Box

Yifu Zhang, Peize Sun+7 more|arXiv (Cornell University)|Oct 13, 2021

Video Surveillance and Tracking Methods85 references105 citations

TL;DR

ByteTrack introduces a simple association strategy that uses almost every detection box, not just high-scoring ones, to improve MOT, achieving state-of-the-art results across multiple benchmarks.

ABSTRACT

Multi-object tracking (MOT) aims at estimating bounding boxes and identities of objects in videos. Most methods obtain identities by associating detection boxes whose scores are higher than a threshold. The objects with low detection scores, e.g. occluded objects, are simply thrown away, which brings non-negligible true object missing and fragmented trajectories. To solve this problem, we present a simple, effective and generic association method, tracking by associating almost every detection box instead of only the high score ones. For the low score detection boxes, we utilize their similarities with tracklets to recover true objects and filter out the background detections. When applied to 9 different state-of-the-art trackers, our method achieves consistent improvement on IDF1 score ranging from 1 to 10 points. To put forwards the state-of-the-art performance of MOT, we design a simple and strong tracker, named ByteTrack. For the first time, we achieve 80.3 MOTA, 77.3 IDF1 and 63.1 HOTA on the test set of MOT17 with 30 FPS running speed on a single V100 GPU. ByteTrack also achieves state-of-the-art performance on MOT20, HiEve and BDD100K tracking benchmarks. The source code, pre-trained models with deploy versions and tutorials of applying to other trackers are released at https://github.com/ifzhang/ByteTrack.

Motivation & Objective

• Motivate alleviating missing detections and fragmented trajectories due to discarding low-score boxes in MOT systems.
• Propose a generic data association method that leverages both high- and low-score detections via tracklet similarity.
• Design a strong, simple tracker (ByteTrack) by pairing BYTE with a high-performance detector.
• Demonstrate ByteTrack’s effectiveness across diverse trackers and standard MOT benchmarks.

Proposed method

• Split detections into high-score and low-score groups using a threshold and apply a two-stage data association.
• First associate high-score detections with existing tracklets using motion/appearance similarity (Kalman filter + IoU or Re-ID features).
• Second associate unmatched tracklets with low-score detections using IoU to recover occluded objects and filter out background.
• Delete unmatched low-score detections and manage lost tracks with a fixed frame horizon (e.g., 30 frames).
• Optionally integrate BYTE with other trackers (e.g., FairMOT) to enhance performance without changing core architectures.

Experimental results

Research questions

• RQ1 Can incorporating low-score detections through a two-stage association reduce missed objects and ID switches in MOT?
• RQ2 Does using a BYTE-based two-stage association improve MOTA/IDF1/HOTA across diverse trackers and datasets?
• RQ3 How robust is the BYTE approach to different detection score thresholds and challenging scenarios (occlusion, motion blur, crowded scenes)?
• RQ4 What is ByteTrack’s performance relative to state-of-the-art methods on MOT17, MOT20, HiEve, and BDD100K under private detector protocols?

Key findings

• Applying BYTE to nine trackers yields consistent improvements in MOTA, IDF1, and IDs across benchmarks.
• ByteTrack achieves 80.3 MOTA, 77.3 IDF1, and 63.1 HOTA on MOT17 test set at 30 FPS on a single V100 GPU.
• On MOT17, ByteTrack ranks first with 30 FPS, outperforming the second-best by notable margins.
• On MOT20, ByteTrack achieves 77.8 MOTA, 75.2 IDF1, and 61.3 HOTA, outperforming competitors in crowded scenes.
• Table reports indicate ByteTrack’s performance gains even when using only IoU for similarity in the second association.
• ByteTrack demonstrates strong generalization when integrated with various trackers and remains robust to detection score thresholds.