[Paper Review] CNN-based Density Estimation and Crowd Counting: A Survey

Guangshuai Gao, Junyu Gao+3 more|arXiv (Cornell University)|Mar 28, 2020

Video Surveillance and Tracking Methods219 references141 citations

TL;DR

This paper surveys CNN-based density estimation and crowd counting models, analyzes architectures, learning paradigms, and evaluation datasets, and provides density maps, test results, and open issues to guide future research.

ABSTRACT

Accurately estimating the number of objects in a single image is a challenging yet meaningful task and has been applied in many applications such as urban planning and public safety. In the various object counting tasks, crowd counting is particularly prominent due to its specific significance to social security and development. Fortunately, the development of the techniques for crowd counting can be generalized to other related fields such as vehicle counting and environment survey, if without taking their characteristics into account. Therefore, many researchers are devoting to crowd counting, and many excellent works of literature and works have spurted out. In these works, they are must be helpful for the development of crowd counting. However, the question we should consider is why they are effective for this task. Limited by the cost of time and energy, we cannot analyze all the algorithms. In this paper, we have surveyed over 220 works to comprehensively and systematically study the crowd counting models, mainly CNN-based density map estimation methods. Finally, according to the evaluation metrics, we select the top three performers on their crowd counting datasets and analyze their merits and drawbacks. Through our analysis, we expect to make reasonable inference and prediction for the future development of crowd counting, and meanwhile, it can also provide feasible solutions for the problem of object counting in other fields. We provide the density maps and prediction results of some mainstream algorithm in the validation set of NWPU dataset for comparison and testing. Meanwhile, density map generation and evaluation tools are also provided. All the codes and evaluation results are made publicly available at https://github.com/gaoguangshuai/survey-for-crowd-counting.

Motivation & Objective

• Provide a comprehensive review of CNN-based density estimation and crowd counting models.
• Classify methods by network architecture, supervision form, learning paradigm, and cross-domain generalization.
• Analyze factors affecting performance and identify top-performing approaches and their merits/drawbacks.
• Present datasets, evaluation metrics, and benchmark results to guide future research.
• Offer open questions and future directions for datasets, models, and generalization to other domains.

Proposed method

• Survey and categorize CNN-based crowd counting models into basic, multi-column, and single-column architectures.
• Examine learning paradigms including single-task and multi-task frameworks.
• Discuss inference manners as patch-based and whole image-based approaches.
• Review supervision forms from instance-level to weakly supervised and multi-task settings.
• Provide density map generation and evaluation tools and publicly share code and results.
• Benchmark representative models on standard datasets and analyze their strengths and limitations.

Experimental results

Research questions

• RQ1 What are the representative CNN-based network architectures for crowd counting and density estimation?
• RQ2 How do learning paradigms and supervision forms influence performance and generalization?
• RQ3 What datasets and evaluation metrics best capture progress in CNN-based crowd counting, and which models top the benchmarks?
• RQ4 What open challenges remain and which directions appear most promising for future work?

Key findings

• The survey systematically reviews CNN-based density estimation crowd counting models and provides taxonomies and analysis.
• It identifies top-performing methods on representative datasets and analyzes their merits and drawbacks.
• Density maps and prediction results for mainstream algorithms are provided on the NWPU validation set for comparison.
• The authors supply density map generation and evaluation tools and publicly release code and results for reproducibility.
• The work discusses open issues, challenges, and future directions in model design, dataset collection, and domain adaptation.
• The paper contextualizes the evolution from basic CNNs to multi-column and single-column architectures with learning paradigms and supervision levels.