[Paper Review] Learning from Synthetic Data for Crowd Counting in the Wild

Qi Wang, Junyu Gao+2 more|arXiv (Cornell University)|Mar 8, 2019

Video Surveillance and Tracking MethodsComputer Science37 references51 citations

TL;DR

The paper creates a large synthetic GCC dataset from GTA5 with automatic labeling and shows two approaches to improve real-world crowd counting: pretraining on synthetic data followed by finetuning, and domain adaptation via an SSIM-embedded CycleGAN to translate synthetic to real imagery for unsupervised learning.

ABSTRACT

Recently, counting the number of people for crowd scenes is a hot topic because of its widespread applications (e.g. video surveillance, public security). It is a difficult task in the wild: changeable environment, large-range number of people cause the current methods can not work well. In addition, due to the scarce data, many methods suffer from over-fitting to a different extent. To remedy the above two problems, firstly, we develop a data collector and labeler, which can generate the synthetic crowd scenes and simultaneously annotate them without any manpower. Based on it, we build a large-scale, diverse synthetic dataset. Secondly, we propose two schemes that exploit the synthetic data to boost the performance of crowd counting in the wild: 1) pretrain a crowd counter on the synthetic data, then finetune it using the real data, which significantly prompts the model's performance on real data; 2) propose a crowd counting method via domain adaptation, which can free humans from heavy data annotations. Extensive experiments show that the first method achieves the state-of-the-art performance on four real datasets, and the second outperforms our baselines. The dataset and source code are available at https://gjy3035.github.io/GCC-CL/.

Motivation & Objective

• Motivate crowd counting in the wild by addressing data scarcity and domain gap between synthetic and real scenes.
• Construct a large, diverse synthetic dataset (GCC) with automatic annotations from GTA5.
• Propose a supervised pretraining-and-finetuning strategy that leverages GCC to improve real-data performance.
• Develop a domain adaptation method (SE Cycle GAN) to translate synthetic to photo-realistic imagery and train without real-data labels.
• Evaluate on multiple real-world datasets to demonstrate performance gains and domain-transfer capabilities.

Proposed method

• Introduce the Spatial Fully Convolutional Network (SFCN) to regress density maps directly from images.
• Create the GTA5 Crowd Counting (GCC) dataset with 15,212 images and 7,625,843 labeled heads across 400 scenes, with diverse weather, times, and locations.
• Pretrain a crowd counter on GCC and finetune on real data to improve initialization and reduce overfitting.
• Propose an SSIM Embedding (SE) Cycle GAN to translate synthetic scenes to photo-realistic images while preserving local textures using an SSIM-based cycle-consistency loss.
• Incorporate Density/Scene Regularization to bound outputs with a MAX_S value and selectively sample translated data for datasets with domain differences.
• Demonstrate that pretraining on GCC yields lower MAE/MSE on real datasets compared to training from scratch or using ImageNet initialization.

Experimental results

Research questions

• RQ1 Can synthetic data pretraining on GCC improve crowd counting performance on real datasets after finetuning?
• RQ2 Does a domain-adaptive translation (SE Cycle GAN) reduce the synthetic-to-real domain gap sufficiently to enable unsupervised training on real data?
• RQ3 What are the gains of using a density-aware normalization and data selection strategy in domain adaptation for crowd counting?
• RQ4 How does the proposed SFCN compare to established baselines under various train/test splits on GCC?
• RQ5 To what extent can synthetic data enable state-of-the-art performance on real crowd counting benchmarks?

Key findings

• Pretraining on GCC followed by finetuning on real data reduces counting errors (MAE/MSE) compared with training from scratch or using ImageNet initializations (e.g., MCNN: from 277/426 to 199.8/311.2 on UCF-QNRF and 26.4/41.3 to 18.8/28.2 on SHHT B).
• SFCN achieves competitive and often superior accuracy on GCC under random, cross-camera, and cross-location splits (e.g., SFCN: 36.2/81.1 MAE/MSE on random; 56.0/129.7 on cross-camera; 89.3/216.8 on cross-location).
• The SFCN† model finetuned on GCC attains state-of-the-art results on five real datasets (e.g., UCF-QNRF: 102.0/171.4 MAE/MSE vs SOTA 132/191).
• SE Cycle GAN substantially improves domain adaptation over Cycle GAN and No Adaptation across multiple real datasets (e.g., on ShanghaiTech A: NoAdpt 160.0 MAE vs CycleGAN 143.3 MAE vs SE Cycle GAN 123.4 MAE).
• Density/Scene Regularization (DSR) improves adaptation by filtering synthetic scenes not present in real datasets, boosting performance over Cycle GAN with and without DSR on ShanghaiTech A.