[Paper Review] Knock, knock. Who's there? -- Identifying football player jersey numbers with synthetic data

Divya Bhargavi, Erika Pelaez Coyotl+1 more|arXiv (Cornell University)|Mar 1, 2022

Video Analysis and Summarization4 citations

TL;DR

This paper proposes a lightweight, two-stage approach for detecting football jersey numbers in low-data, highly imbalanced datasets using synthetic data augmentation and pose-guided localization. By leveraging pretrained models for person detection and human pose estimation to crop torso regions, and training lightweight CNNs on synthetic datasets (Simple2D and Complex2D), the method achieves 89% accuracy—representing a 9% improvement over baseline models—demonstrating state-of-the-art performance with minimal annotation effort.

ABSTRACT

Automatic player identification is an essential and complex task in sports video analysis. Different strategies have been devised over the years, but identification based on jersey numbers is one of the most common approaches given its versatility and relative simplicity. However, automatic detection of jersey numbers is still challenging due to changing camera angles, low video resolution, small object size in wide-range shots and transient changes in the player's posture and movement. In this paper we present a novel approach for jersey number identification in a small, highly imbalanced dataset from the Seattle Seahawks practice videos. Our results indicate that simple models can achieve an acceptable performance on the jersey number detection task and that synthetic data can improve the performance dramatically (accuracy increase of ~9% overall, ~18% on low frequency numbers) making our approach achieve state of the art results.

Motivation & Objective

• To address the challenge of low-data, highly imbalanced datasets in football jersey number detection.
• To reduce annotation burden by eliminating the need for bounding box annotations for number locations.
• To improve model generalization and performance in low-resource settings using synthetic data.
• To develop a scalable, minimal-supervision pipeline applicable to sports video analysis with limited real-world data.
• To demonstrate that lightweight models with synthetic pretraining can achieve state-of-the-art results in jersey number detection.

Proposed method

• A three-step pipeline begins with a pretrained person detection model to localize and crop players in video frames.
• A pretrained human pose estimation model identifies torso key-points to localize the jersey region, avoiding manual bounding box annotation for numbers.
• Cropped torso images (20×25px on average) are fed into two lightweight CNNs: one for multi-class two-digit number recognition and one for multi-label digit-wise detection.
• Two synthetic dataset generators—Simple2D (fonts and colors) and Complex2D (superimposed on COCO images)—are used to augment training data and improve robustness.
• Models are pre-trained on synthetic data and fine-tuned on real-world Seattle Seahawks practice videos to improve generalization.
• An ensemble of multi-class and multi-label models achieves the final 89% accuracy.

Experimental results

Research questions

• RQ1 Can synthetic data generation significantly improve performance in low-data, imbalanced jersey number detection tasks?
• RQ2 Can pose estimation be used effectively to localize jersey numbers without bounding box annotations?
• RQ3 Does pretraining on synthetic data lead to measurable improvements in accuracy, especially for low-frequency numbers?
• RQ4 Can lightweight CNNs achieve state-of-the-art performance when combined with synthetic data and pose-guided cropping?
• RQ5 To what extent does synthetic data enhance generalization in highly imbalanced real-world sports video datasets?

Key findings

• The ensemble model achieved 89% accuracy on the test set, representing a 9% improvement over baseline models trained only on real data.
• The use of synthetic data led to an 18% accuracy increase on low-frequency jersey numbers, demonstrating strong generalization for rare classes.
• Pretraining on Complex2D synthetic data yielded the highest performance, with 83.06% accuracy on the ensemble model.
• The method achieved state-of-the-art performance without modifying the data collection pipeline or requiring additional hardware sensors.
• Image quality and resolution of the cropped input were found to be critical factors, with confidence directly correlated to input clarity.
• No improvement was observed from data duplication or standard image augmentation, highlighting the value of high-quality synthetic data over simple augmentation.