[Paper Review] CrowdNet: A Deep Convolutional Network for Dense Crowd Counting

Lokesh Boominathan, Srinivas S S Kruthiventi+1 more|arXiv (Cornell University)|Aug 22, 2016

Video Surveillance and Tracking MethodsComputer Science17 references122 citations

TL;DR

CrowdNet combines deep and shallow fully convolutional networks to predict dense crowd density maps from images, using multi-scale data augmentation to handle scale variations, and achieves state-of-the-art MAE on UCF_CC_50.

ABSTRACT

Our work proposes a novel deep learning framework for estimating crowd density from static images of highly dense crowds. We use a combination of deep and shallow, fully convolutional networks to predict the density map for a given crowd image. Such a combination is used for effectively capturing both the high-level semantic information (face/body detectors) and the low-level features (blob detectors), that are necessary for crowd counting under large scale variations. As most crowd datasets have limited training samples (<100 images) and deep learning based approaches require large amounts of training data, we perform multi-scale data augmentation. Augmenting the training samples in such a manner helps in guiding the CNN to learn scale invariant representations. Our method is tested on the challenging UCF_CC_50 dataset, and shown to outperform the state of the art methods.

Motivation & Objective

• Motivate accurate crowd density estimation in highly dense scenes from still images.
• Develop a network that leverages both high-level semantic cues and low-level blob patterns.
• Address limited training data via multi-scale data augmentation.
• Produce dense density maps and total crowd counts for analysis and safety applications.

Proposed method

• Use a deep CNN (VGG-16 like) stripped of fully connected layers to make per-pixel density predictions at 1/8 resolution.
• Use a shallow 3-layer CNN to detect small head-blobs and complement deep features.
• Concatenate deep and shallow predictions via a 1x1 convolution and upsample to input size for final density map.
• Train using ground truth created by Gaussian-blurred head annotations to preserve total count.
• Augment training with multi-scale image patches (scales 0.5 to 1.2) and oversample high-density patches to handle scale variation and crowd density.

Experimental results

Research questions

• RQ1 Can a hybrid deep+shallow CNN accurately predict crowd density maps in extremely dense scenes?
• RQ2 Does multi-scale data augmentation improve robustness to scale variation and occlusion in crowd counting?
• RQ3 Is the generated Gaussian-ground-truth effective for training per-pixel density estimation models?
• RQ4 What is the impact of combining deep and shallow representations on counting accuracy?

Key findings

• Achieves state-of-the-art MAE (452.5) on UCF_CC_50 compared to prior methods.
• Combining deep and shallow networks yields better MAE (645) than either network alone (Deep: 681, Shallow: 1107).
• Data augmentation targeting dense regions nearly doubles training patches (26,385 to 50,891) and reduces MAE (from 725 to 645).
• The model can estimate counts close to actual in most images, though it underestimates in extremely dense cases (>2500 people).
• Density maps and total counts are obtained by summing the predicted density and the network is trained with L2 loss.