[Paper Review] BoT-SORT: Robust Associations Multi-Pedestrian Tracking

Nir Aharon, Roy Orfaig+1 more|arXiv (Cornell University)|Jun 29, 2022

Video Surveillance and Tracking Methods308 citations

TL;DR

BoT-SORT introduces camera-motion compensation and a refined Kalman filter state along with an IoU-ReID fusion to achieve state-of-the-art multi-pedestrian tracking on MOT17/20, outperforming prior trackers in MOTA, IDF1, and HOTA.

ABSTRACT

The goal of multi-object tracking (MOT) is detecting and tracking all the objects in a scene, while keeping a unique identifier for each object. In this paper, we present a new robust state-of-the-art tracker, which can combine the advantages of motion and appearance information, along with camera-motion compensation, and a more accurate Kalman filter state vector. Our new trackers BoT-SORT, and BoT-SORT-ReID rank first in the datasets of MOTChallenge [29, 11] on both MOT17 and MOT20 test sets, in terms of all the main MOT metrics: MOTA, IDF1, and HOTA. For MOT17: 80.5 MOTA, 80.2 IDF1, and 65.0 HOTA are achieved. The source code and the pre-trained models are available at https://github.com/NirAharon/BOT-SORT

Motivation & Objective

• Improve multi-object tracking by integrating motion, appearance, and camera-motion compensation to handle dynamic scenes.
• Develop a robust Kalman filter state vector for better bounding box localization.
• Fuse IoU-based motion cues with Re-ID appearance features for robust data association.

Proposed method

• Use a Kalman filter with a revised state vector capturing center coordinates, width, height, and their velocities (x_c, y_c, w, h, ẋ_c, ẏ_c, ẇ, ḣ).
• Incorporate Camera Motion Compensation (CMC) via image registration with affine transforms to estimate background motion and adjust KF predictions.
• Introduce a new IoU-ReID fusion cost that first applies proximity masking and then takes the minimum of IoU distance and a refined cosine distance, controlled by thresholds θ_iou and θ_emb.
• Adopt an EMA update for tracklet appearance descriptors f_i^k into e_i^k to stabilize Re-ID features, using a high-confidence detection filter.
• Combine motion (IoU) and appearance (cosine) cues in a cost matrix C used in the Hungarian assignment for the first association step.
• Provide an online TB (BoT-SORT) and a ReID-extended version (BoT-SORT-ReID) with a simple yet effective integration into the ByteTrack framework.

Experimental results

Research questions

• RQ1 How can camera motion be effectively compensated to improve tracklet-to-detection associations in MOT?
• RQ2 Does a refined Kalman filter state vector improve bounding box localization and overall tracking metrics?
• RQ3 Can IoU-based motion cues be effectively fused with Re-ID features to enhance data association without compromising speed?
• RQ4 What is the impact of appearance integration via EMA-reported features on identity preservation in crowded scenes?

Key findings

• BoT-SORT and BoT-SORT-ReID achieve top metrics on MOT17 and MOT20 test sets, ranking first in MOTA, IDF1, and HOTA in leadboards.
• On MOT17 test set, BoT-SORT achieves MOTA 80.6, IDF1 79.5, and HOTA 64.6 with 1257 IDs and 6.6 FPS.
• BoT-SORT-ReID achieves MOTA 80.5, IDF1 80.2, and HOTA 65.0 with 1212 IDs and 4.5 FPS.
• Ablation shows that adding KF, CMC, and ReID components yields progressive improvements in MOTA, IDF1, and HOTA, with the full BoT-SORT setup performing best.
• IoU-ReID fusion with masking (proximity thresholds) outperforms individual IoU or Cosine strategies in MOTA, IDF1, and HOTA on MOT17 validation.
• The paper also introduces cMOTA, a time-resolved MOTA metric for tracking failure analysis.