[Paper Review] Crowd Counting by Adaptively Fusing Predictions from an Image Pyramid

Di Kang, Antoni B. Chan|arXiv (Cornell University)|May 16, 2018

Video Surveillance and Tracking MethodsComputer Science18 references94 citations

TL;DR

The paper proposes an image-pyramid based crowd counting approach that adaptively fuses density predictions from multiple scales using an across-scale attention map and 1x1 fusion, achieving fast, near real-time performance with strong accuracy.

ABSTRACT

Because of the powerful learning capability of deep neural networks, counting performance via density map estimation has improved significantly during the past several years. However, it is still very challenging due to severe occlusion, large scale variations, and perspective distortion. Scale variations (from image to image) coupled with perspective distortion (within one image) result in huge scale changes of the object size. Earlier methods based on convolutional neural networks (CNN) typically did not handle this scale variation explicitly, until Hydra-CNN and MCNN. MCNN uses three columns, each with different filter sizes, to extract features at different scales. In this paper, in contrast to using filters of different sizes, we utilize an image pyramid to deal with scale variations. It is more effective and efficient to resize the input fed into the network, as compared to using larger filter sizes. Secondly, we adaptively fuse the predictions from different scales (using adaptively changing per-pixel weights), which makes our method adapt to scale changes within an image. The adaptive fusing is achieved by generating an across-scale attention map, which softly selects a suitable scale for each pixel, followed by a 1x1 convolution. Extensive experiments on three popular datasets show very compelling results.

Motivation & Objective

• Address large intra-image scale variation and perspective distortion in crowd counting.
• Propose an image-pyramid based FCN backbone to generate scale-specific density maps.
• Develop an adaptive fusion mechanism with across-scale attention to per-pixel select appropriate scales.
• Demonstrate state-of-the-art or competitive performance with real-time or faster-than-real-time inference.
• Evaluate on ShanghaiTech, WorldExpo, and UCSD datasets to validate effectiveness.

Proposed method

• Construct an image pyramid by downsampling the input image to multiple scales.
• Process each scale with a shared backbone FCN to produce a density map.
• Generate an across-scale attention map from the last feature map of the scale-specific branch.
• Apply across-scale softmax to obtain per-pixel scale weights and multiply with corresponding density maps.
• Fuse the rectified density maps from all scales using a 1x1 convolution to obtain the final density map.
• Train end-to-end with per-pixel MSE loss on 32x32 density patches (from 128x128 inputs).

Experimental results

Research questions

• RQ1 Can an image pyramid with adaptive per-pixel scale fusion improve crowd counting under varying object sizes and perspective within a single image?
• RQ2 Does attention-guided fusion outperform fixed or simple fusion strategies for multi-scale density maps?
• RQ3 How does the proposed FCN backbone with limited downsampling affect density map quality and runtime?
• RQ4 What is the method's performance relative to existing multi-scale counting approaches on standard datasets?

Key findings

• Adaptive image-pyramid fusion improves over single-scale FCN baselines on ShanghaiTech Part A/B, WorldExpo, and UCSD.
• FCN-7c-3s (3-scale fusion) achieves MAE 80.6 and RMSE 126.7 on ShanghaiTech Part A, MAE 10.2 and RMSE 18.3 on Part B.
• Our method with 2 scales (FCN-7c-2s) achieves MAE 81.3 and RMSE 132.6 on Part A, MAE 10.9 and RMSE 19.1 on Part B.
• Compared to CNN-patch, MCNN, Switch-CNN, and CP-CNN, our FCN-7c-3s has competitive MAE/MSE while offering faster-than-real-time inference on high-resolution images.
• The attention-based fusion (across-scale softmax) is crucial; ablations without softmax or with fixed fusion underperform.
• Our approach runs at 158-439 fps depending on configuration, offering a favorable speed-accuracy balance.