[Paper Review] SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes

Yutao Cui, Chenkai Zeng+4 more|arXiv (Cornell University)|Apr 11, 2023

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper introduces SportsMOT, a large-scale multi-object tracking dataset comprising 240 video sequences across basketball, volleyball, and football, with over 150K frames and 1.6M bounding boxes. To address challenges in motion and appearance-based association, the authors propose MixSort, a tracker integrating a MixFormer-like appearance model with motion-based tracking, achieving state-of-the-art performance with 73.8 HOTA on SportsMOT and improved generalization over MOT17 and SoccerNet.

ABSTRACT

Multi-object tracking in sports scenes plays a critical role in gathering players statistics, supporting further analysis, such as automatic tactical analysis. Yet existing MOT benchmarks cast little attention on the domain, limiting its development. In this work, we present a new large-scale multi-object tracking dataset in diverse sports scenes, coined as \emph{SportsMOT}, where all players on the court are supposed to be tracked. It consists of 240 video sequences, over 150K frames (almost 15 imes MOT17) and over 1.6M bounding boxes (3 imes MOT17) collected from 3 sports categories, including basketball, volleyball and football. Our dataset is characterized with two key properties: 1) fast and variable-speed motion and 2) similar yet distinguishable appearance. We expect SportsMOT to encourage the MOT trackers to promote in both motion-based association and appearance-based association. We benchmark several state-of-the-art trackers and reveal the key challenge of SportsMOT lies in object association. To alleviate the issue, we further propose a new multi-object tracking framework, termed as \emph{MixSort}, introducing a MixFormer-like structure as an auxiliary association model to prevailing tracking-by-detection trackers. By integrating the customized appearance-based association with the original motion-based association, MixSort achieves state-of-the-art performance on SportsMOT and MOT17. Based on MixSort, we give an in-depth analysis and provide some profound insights into SportsMOT. The dataset and code will be available at https://deeperaction.github.io/datasets/sportsmot.html.

Motivation & Objective

• Address the lack of large-scale, sports-specific multi-object tracking benchmarks tailored for player tracking in dynamic, high-speed environments.
• Identify key challenges in sports MOT, particularly in motion-based and appearance-based association due to fast and variable-speed player movements and visually similar jerseys.
• Develop a new tracking framework, MixSort, that enhances both motion and appearance modeling to improve association accuracy in complex sports scenes.
• Provide a comprehensive benchmark to evaluate and advance multi-object tracking models specifically for sports applications.
• Enable deeper analysis of tracker generalization across sports categories and comparison with existing datasets like MOT17, DanceTrack, and SoccerNet.

Proposed method

• Collect and annotate 240 high-quality video sequences from basketball, volleyball, and football, covering over 150K frames and 1.6 million bounding boxes, with all players on the court tracked.
• Design a new multi-object tracking framework, MixSort, that integrates a MixFormer-like structure as an auxiliary association module to enhance appearance-based tracking.
• Combine motion-based association (using Kalman filter and IoU) with appearance-based association (via the MixFormer module) using a learnable fusion mechanism.
• Fine-tune the ReID model on SportsMOT to improve discriminative feature learning for similar-looking players with distinct numbers and postures.
• Train and evaluate multiple state-of-the-art trackers (e.g., ByteTrack, DeepSORT) on SportsMOT, using a train/val/test split of 45/45/150 sequences.
• Use a weighted fusion strategy (hyperparameter α) to combine IoU and MixSort-based association scores, optimizing for HOTA on validation.

Figure 1 : Sampled sequences from the categories of basketball and football of SportsMOT, MOT17 and DanceTrack. There exist two key properties of SportsMOT: 1) fast and variable-speed motion , i.e . players usually possess high speed and frequently change their running speed (the visualized adjacent

Experimental results

Research questions

• RQ1 How do existing state-of-the-art multi-object trackers perform on a large-scale sports-specific MOT benchmark with high-speed and variable-motion players?
• RQ2 What are the dominant failure modes in current trackers when applied to sports scenes, particularly regarding motion modeling and appearance discrimination?
• RQ3 To what extent can a MixFormer-like architecture improve appearance-based association in sports MOT, especially given the challenge of similar yet distinguishable appearances?
• RQ4 How does SportsMOT compare to existing benchmarks like MOT17, DanceTrack, and SoccerNet in terms of tracking difficulty and model generalization?
• RQ5 Can a hybrid motion-appearance association framework outperform standard trackers that rely on either motion or appearance alone in complex sports environments?

Key findings

• SportsMOT presents a significantly more challenging benchmark than MOT17, with lower IDF1 and AssA scores (67.8 and 46.8 on basketball) despite high detection accuracy (79.1 DetA), indicating that association is the primary bottleneck.
• The MixSort framework achieves a state-of-the-art HOTA of 73.8 on SportsMOT, outperforming standard IoU-based association (71.5 HOTA) and demonstrating the effectiveness of the MixFormer-based appearance model.
• On the basketball subset, MixSort achieves 60.8 HOTA, the lowest among sports categories, highlighting the difficulty of fast, high-velocity player motion and occlusions.
• MixSort outperforms both standard ByteTrack and ByteTrack with ReID on SportsMOT, improving HOTA by 0.9, IDF1 by 1.9, and AssA by 1.4, proving the superiority of the proposed appearance modeling.
• On DanceTrack, MixSort underperforms compared to standard ByteTrack, indicating that SportsMOT’s appearance cues are more discriminative than DanceTrack’s, which features indistinguishable costumes, thus validating SportsMOT’s dual challenge in motion and appearance.
• SoccerNet, which focuses only on football, shows lower DetA (71.5) and HOTA (62.9) than SportsMOT (78.8 DetA, 65.7 HOTA), suggesting that SportsMOT’s broader scope and higher-quality annotations lead to better tracker performance and more robust evaluation.

Figure 2 : IoU on adjacent frames. (a) Compared to MOT17 and DanceTrack, SportsMOT has a lower score, indicating that objects have faster motion. (b) In SportsMOT, the category of football has the lowest IoU score, which means that football players often have fast motion.