[Paper Review] Sports Camera Calibration via Synthetic Data

Jianhui Chen, James J. Little|arXiv (Cornell University)|Oct 25, 2018

Human Pose and Action Recognition24 references5 citations

TL;DR

This paper proposes a highly automatic sports camera calibration method using synthetic data and deep learning. It introduces a novel 3-parameter camera pose engine to generate synthetic edge images, trains a siamese network for compact feature learning, and employs a two-GAN model for robust field marking detection; the method achieves state-of-the-art accuracy on a standard soccer dataset and 97.6% mean IoU on a volleyball dataset.

ABSTRACT

Calibrating sports cameras is important for autonomous broadcasting and sports analysis. Here we propose a highly automatic method for calibrating sports cameras from a single image using synthetic data. First, we develop a novel camera pose engine. The camera pose engine has only three significant free parameters so that it can effectively generate a lot of camera poses and corresponding edge (i.e, field marking) images. Then, we learn compact deep features via a siamese network from paired edge image and camera pose and build a feature-pose database. After that, we use a novel two-GAN (generative adversarial network) model to detect field markings in real images. Finally, we query an initial camera pose from the feature-pose database and refine camera poses using truncated distance images. We evaluate our method on both synthetic and real data. Our method not only demonstrates the robustness on the synthetic data but also achieves the state-of-the-art accuracy on a standard soccer dataset and very high performance on a volleyball dataset.

Motivation & Objective

• To develop a fully automatic camera calibration method for sports videos that minimizes human intervention.
• To address the challenge of camera pose estimation from a single image in diverse sports environments with varying camera locations and field markings.
• To leverage synthetic data and deep learning to improve calibration robustness and accuracy across different sports stadiums and lighting conditions.
• To exploit the spatial and angular priors of sports camera placements (e.g., near the middle line, limited tilt) to simplify the pose estimation problem.
• To achieve high-accuracy calibration on real-world datasets, particularly in scenarios with limited or uneven field markings.

Proposed method

• A novel camera pose engine with only three significant free parameters (pan, tilt, focal length) is designed to generate diverse camera poses and corresponding synthetic edge images of sports fields.
• A siamese deep network is trained to learn compact 16-dimensional features from paired edge images and camera poses, enabling efficient feature-pose database construction.
• A two-GAN model is introduced: one GAN performs instance segmentation of field markings, and another performs dense optical flow-like warping (LK-based) to refine detection.
• A feature-pose database is built by mapping learned features to their corresponding camera poses for fast nearest-neighbor retrieval during inference.
• Initial camera pose is retrieved from the database and refined using truncated distance images, which encode geometric constraints from detected field markings.
• The method uses a hybrid approach: synthetic data for training, real image detection via GANs, and geometric refinement to improve accuracy.

Experimental results

Research questions

• RQ1 Can a highly automatic camera calibration system be built using only a single image and synthetic data, without requiring manual annotations during inference?
• RQ2 How can the spatial and angular priors of sports camera placements (e.g., location near the middle line, limited tilt) be leveraged to simplify and improve calibration accuracy?
• RQ3 To what extent can a siamese network with compact features (16D) generalize to real-world camera pose retrieval from edge images?
• RQ4 Can a two-GAN architecture effectively detect sparse and irregular field markings in real images, even under challenging conditions like shadows or partial visibility?
• RQ5 How does the proposed method compare in accuracy and robustness to existing state-of-the-art methods on standard and non-standard sports datasets?

Key findings

• The method achieves state-of-the-art performance on the standard soccer dataset, outperforming prior methods in camera pose estimation accuracy.
• On the volleyball dataset, the method achieves a mean IoU of 97.6% and a median IoU of 98.8%, demonstrating very high performance despite fewer visible markings.
• The two-GAN model significantly improves field marking detection, with ablation studies showing both the segmentation GAN and LK-based warping contribute to final accuracy.
• The method remains robust even when field markings are partially occluded or unevenly distributed, though heavy shadows degrade detection and thus pose estimation.
• The HOG feature baseline performs comparably in accuracy but is far less compact (1,860D vs. 16D), highlighting the efficiency of the proposed siamese network.
• The method is robust to camera location variations across stadiums, as it only assumes a rough prior on camera position, unlike methods requiring exact known locations.