[Paper Review] Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowds

Haroon Idrees, Muhmmad Tayyab+5 more|arXiv (Cornell University)|Aug 2, 2018

Video Surveillance and Tracking Methods22 references61 citations

TL;DR

Introduces Composition Loss to jointly train a CNN for counting, density map estimation, and localization in dense crowds and releases the large UCF-QNRF dataset; shows state-of-the-art results across counting, density, and localization tasks.

ABSTRACT

With multiple crowd gatherings of millions of people every year in events ranging from pilgrimages to protests, concerts to marathons, and festivals to funerals; visual crowd analysis is emerging as a new frontier in computer vision. In particular, counting in highly dense crowds is a challenging problem with far-reaching applicability in crowd safety and management, as well as gauging political significance of protests and demonstrations. In this paper, we propose a novel approach that simultaneously solves the problems of counting, density map estimation and localization of people in a given dense crowd image. Our formulation is based on an important observation that the three problems are inherently related to each other making the loss function for optimizing a deep CNN decomposable. Since localization requires high-quality images and annotations, we introduce UCF-QNRF dataset that overcomes the shortcomings of previous datasets, and contains 1.25 million humans manually marked with dot annotations. Finally, we present evaluation measures and comparison with recent deep CNN networks, including those developed specifically for crowd counting. Our approach significantly outperforms state-of-the-art on the new dataset, which is the most challenging dataset with the largest number of crowd annotations in the most diverse set of scenes.

Motivation & Objective

• Motivate accurate counting in very dense crowds for safety and security applications.
• Propose a joint learning framework that decomposes counting, density estimation, and localization losses.
• Create and annotate a large, high-quality dataset (UCF-QNRF) tailored for dense crowds.
• Demonstrate that density and localization supervision improves counting performance on diverse scenes.

Proposed method

• Define a decomposable Composition Loss that ties together counting, density maps, and localization via adaptive Gaussian kernels.
• Branch a Density Network from a DenseNet base to output multiple density levels (D1, D2) and a localization map (Dinfty).
• Compute densities with per-person adaptive bandwidth sigma_i = min(distance to nearest neighbor, tau) and generate a sequence of density maps D_k with f_k(sigma) = sigma^{1/k}.
• Train with losses L_c (count regression) and L_k (MSE between predicted and ground truth density/localization maps) across multiple density levels, enforcing that their counts align with the true count.
• Utilize DenseNet-201 as the backbone and attach the Density Network blocks to DenseBlock2 to predict D1, D2, and Dinfty with intermediate supervision.

Experimental results

Research questions

• RQ1 Can counting, density estimation, and localization be trained jointly without loss of performance degradation?
• RQ2 Does composing multiple density levels with adaptive kernels improve localization accuracy and density map quality?
• RQ3 What is the impact of the Composition Loss on counting accuracy compared to single-task or multi-task baselines?
• RQ4 Does the proposed large-scale UCF-QNRF dataset enable better generalization for dense crowd analysis?

Key findings

• The proposed method achieves a counting MAE of 132, NAE of 0.258, and MSE of 191 on the UCF-QNRF dataset, outperforming several state-of-the-art methods.
• Density map estimation with the proposed loss yields DM-MAE of 0.00044, DM-MSE of 0.0017, and DM-HI of 0.9131, surpassing competing approaches by a large margin.
• Localization results show the proposed approach achieving an average precision of 75.8%, average recall of 59.75%, and L-AUC of 0.714, higher than several baselines.
• Ablation studies confirm that multiple density levels (D1, D2, Dinfty) and the Composition Loss consistently improve counting, density, and localization metrics over single-branch or non-compositional configurations.
• Using intermediate supervision from density and localization maps helps training converge faster and improves performance across tasks.