[Paper Review] Perspective-Guided Convolution Networks for Crowd Counting

Zhaoyi Yan, Yuchen Yuan+5 more|arXiv (Cornell University)|Sep 16, 2019

Video Surveillance and Tracking Methods26 references34 citations

TL;DR

PGCNet uses perspective-guided convolution to adapt feature smoothing to intra-scene scale variations, with a perspective estimation branch and a new Crowd Surveillance dataset, achieving state-of-the-art results on multiple benchmarks.

ABSTRACT

In this paper, we propose a novel perspective-guided convolution (PGC) for convolutional neural network (CNN) based crowd counting (i.e. PGCNet), which aims to overcome the dramatic intra-scene scale variations of people due to the perspective effect. While most state-of-the-arts adopt multi-scale or multi-column architectures to address such issue, they generally fail in modeling continuous scale variations since only discrete representative scales are considered. PGCNet, on the other hand, utilizes perspective information to guide the spatially variant smoothing of feature maps before feeding them to the successive convolutions. An effective perspective estimation branch is also introduced to PGCNet, which can be trained in either supervised setting or weakly-supervised setting when the branch has been pre-trained. Our PGCNet is single-column with moderate increase in computation, and extensive experimental results on four benchmark datasets show the improvements of our method against the state-of-the-arts. Additionally, we also introduce Crowd Surveillance, a large scale dataset for crowd counting that contains 13,000+ high-resolution images with challenging scenarios.

Motivation & Objective

• Address extreme intra-scene scale variation in crowd counting caused by perspective.
• Introduce a learnable perspective-guided convolution (PGC) module to adapt receptive fields spatially.
• Incorporate a perspective estimation branch enabling end-to-end training with or without perspective annotations.
• Propose Crowd Surveillance, a large high-resolution crowd counting dataset for robust benchmarking.
• Demonstrate state-of-the-art results on multiple public datasets.

Proposed method

• Introduce Perspective-Guided Convolution (PGC) that combines spatially variant Gaussian smoothing with a subsequent spatially invariant convolution.
• Compute a blur map from a learnable perspective map using a sigmoid-based normalization and a learned linear relation to standard deviations.
• Approximate spatially variant smoothing efficiently via PCA-based decomposition of Gaussian kernels to reduce computation.
• Embed PGC blocks into a single-column backbone to form PGCNet, stacking five PGC blocks for density map estimation.
• Add a perspective estimation branch (PENet) trained via a three-phase auto-encoder strategy to enable supervised or weakly-supervised learning, enabling end-to-end training even without perspective annotations.
• Introduce the Crowd Surveillance dataset (13,945 high-resolution images) with ROI masks to advance crowd counting research.

Experimental results

Research questions

• RQ1 How can perspective information be leveraged to handle continuous intra-scene crowd scale variations in a single-column CNN?
• RQ2 Can a perspective estimation branch be trained to support end-to-end crowd counting without perspective annotations?
• RQ3 What are the gains of inserting perspective-guided convolutions compared to multi-scale or dilated architectures?
• RQ4 How does the proposed dataset Crowd Surveillance compare to existing benchmarks in terms of resolution and variability?

Key findings

• PGCNet achieves state-of-the-art or competitive MAE/MSE on ShanghaiTech Part A (57.0 MAE, 86.0 MSE) and Part B (8.8 MAE, 13.7 MSE).
• On WorldExpo’10, PGCNet attains the best average MAE (8.1) across scenes with strong improvements over baselines.
• On UCF_CC_50, PGCNet variants reach MAE as low as 244.6 (with end-to-end PENet training) compared to the CSRNet baseline (~264.0).
• On Crowd Surveillance, PGCNet achieves MAE of 7.2 with end-to-end training (vs. 9.8–16.4 range for some baselines).
• The PENet perspective estimator provides robust perspective maps, and end-to-end training with PENet yields additional MAE gains over PENet-guided pre-training alone.
• Crowd Surveillance offers 13,945 high-resolution images and 386k+ labeled people, significantly expanding scale and challenging conditions for crowd counting benchmarks.