[Paper Review] Revisiting Perspective Information for Efficient Crowd Counting

Miaojing Shi, Zhaohui Yang+2 more|arXiv (Cornell University)|Jul 5, 2018

Video Surveillance and Tracking Methods53 references15 citations

TL;DR

This paper proposes a perspective-aware convolutional neural network (PACNN) for efficient crowd counting by integrating learned perspective maps into multi-scale density regression. By generating ground-truth perspective maps and using them as differentiable, learnable weighting layers, PACNN adaptively fuses multi-scale density predictions, achieving state-of-the-art accuracy and inference speed on ShanghaiTech, WorldExpo’10, UCF_CC_50, and UCSD datasets with MAE as low as 0.89 on UCSD.

ABSTRACT

Crowd counting is the task of estimating people numbers in crowd images. Modern crowd counting methods employ deep neural networks to estimate crowd counts via crowd density regressions. A major challenge of this task lies in the perspective distortion, which results in drastic person scale change in an image. Density regression on the small person area is in general very hard. In this work, we propose a perspective-aware convolutional neural network (PACNN) for efficient crowd counting, which integrates the perspective information into density regression to provide additional knowledge of the person scale change in an image. Ground truth perspective maps are firstly generated for training; PACNN is then specifically designed to predict multi-scale perspective maps, and encode them as perspective-aware weighting layers in the network to adaptively combine the outputs of multi-scale density maps. The weights are learned at every pixel of the maps such that the final density combination is robust to the perspective distortion. We conduct extensive experiments on the ShanghaiTech, WorldExpo'10, UCF_CC_50, and UCSD datasets, and demonstrate the effectiveness and efficiency of PACNN over the state-of-the-art.

Motivation & Objective

• To address perspective distortion in crowd counting, which causes drastic scale variation and hinders accurate density regression on small person regions.
• To enable efficient and accurate crowd counting without relying on external camera calibration or scene geometry annotations.
• To learn perspective maps end-to-end during training and use them as adaptive, differentiable weighting layers in the network.
• To improve multi-scale density map fusion by incorporating perspective-aware attention that accounts for person scale changes.
• To achieve state-of-the-art performance with fast inference, especially in perspective-distorted scenes.

Proposed method

• Ground-truth perspective maps are generated by sampling person-scale relations and fitting a nonlinear function based on perspective geometry per image.
• A deep CNN is trained to predict perspective maps end-to-end from input images, enabling test-time inference without external annotations.
• Two perspective-aware weighting layers are introduced, where pixel-wise weights are learned via a nonlinear transformation of predicted perspective maps at different resolutions.
• Multi-scale density maps are adaptively combined using these learned weights, making the final density prediction robust to perspective distortion.
• The network is trained with a multi-task loss combining density regression and perspective map prediction to improve generalization.
• The method supports image-based inference, enabling fast inference (e.g., 230ms for 1024×768 input) without patch-based processing.

Experimental results

Research questions

• RQ1 Can perspective information be effectively learned and integrated into deep neural networks for crowd counting without requiring camera calibration or scene geometry?
• RQ2 How does learning perspective maps improve multi-scale density fusion under severe perspective distortion?
• RQ3 Can a perspective-aware weighting mechanism outperform standard multi-scale fusion in terms of accuracy and inference speed?
• RQ4 Does the proposed method generalize across datasets with varying crowd densities and perspective distortions?
• RQ5 Can the integration of perspective maps lead to state-of-the-art performance with minimal computational overhead?

Key findings

• PACNN achieves the lowest MAE of 0.89 and MSE of 1.18 on the UCSD dataset, outperforming all prior methods.
• On UCF_CC_50, PACNN achieves MAE 241.7 and MSE 320.7 when combined with the [17] backbone, setting a new SOTA.
• On WorldExpo’10, PACNN achieves the lowest mean MAE of 7.8 across all five scenes, demonstrating robustness to perspective variation.
• On ShanghaiTech, PACNN achieves MAE 62.4 and MSE 102.0, surpassing previous SOTA with image-based inference.
• Inference time is only 230ms for a 1024×768 image, significantly faster than patch-based methods (e.g., 5× faster).
• The method generalizes well across diverse datasets, including sparse (UCSD) and dense (ShanghaiTech) crowd scenes.