[Paper Review] Crowd Counting on Images with Scale Variation and Isolated Clusters

Haoyue Bai, Wen Song+1 more|arXiv (Cornell University)|Sep 9, 2019

Video Surveillance and Tracking Methods35 references4 citations

TL;DR

This paper proposes SACANet, a scale-adaptive, long-range context-aware network for crowd counting that addresses large-scale variation and isolated small clusters through a pyramid contextual module, scale-adaptive self-attention multi-branch architecture, and hierarchical fusion. It achieves state-of-the-art performance with significantly lower MAE on VisDrone2019 and ShanghaiTech benchmarks, especially in extreme crowd conditions with diverse scales and scattered clusters.

ABSTRACT

Crowd counting is to estimate the number of objects (e.g., people or vehicles) in an image of unconstrained congested scenes. Designing a general crowd counting algorithm applicable to a wide range of crowd images is challenging, mainly due to the possibly large variation in object scales and the presence of many isolated small clusters. Previous approaches based on convolution operations with multi-branch architecture are effective for only some narrow bands of scales and have not captured the long-range contextual relationship due to isolated clustering. To address that, we propose SACANet, a novel scale-adaptive long-range context-aware network for crowd counting. SACANet consists of three major modules: the pyramid contextual module which extracts long-range contextual information and enlarges the receptive field, a scale-adaptive self-attention multi-branch module to attain high scale sensitivity and detection accuracy of isolated clusters, and a hierarchical fusion module to fuse multi-level self-attention features. With group normalization, SACANet achieves better optimality in the training process. We have conducted extensive experiments using the VisDrone2019 People dataset, the VisDrone2019 Vehicle dataset, and some other challenging benchmarks. As compared with the state-of-the-art methods, SACANet is shown to be effective, especially for extremely crowded conditions with diverse scales and scattered clusters, and achieves much lower MAE as compared with baselines.

Motivation & Objective

• Address the challenge of large-scale variation in crowd counting, where objects of the same physical size appear differently due to perspective and distance.
• Overcome the limitation of local receptive fields in capturing long-range contextual information for isolated small clusters of objects.
• Design a model that maintains high accuracy across diverse scales and scattered clusters, especially in extremely crowded scenes.
• Improve density map regression performance by integrating scale-adaptive attention and hierarchical feature fusion.
• Achieve better generalization and lower error (MAE/MSE) than existing methods on challenging benchmarks with extreme scale variation and isolated clusters.

Proposed method

• Employ a pyramid contextual module to extract long-range contextual features and expand the receptive field using dilated convolutions and multi-scale feature aggregation.
• Introduce a scale-adaptive self-attention mechanism that dynamically selects the most appropriate convolutional branches based on object scale, enhancing sensitivity to small and distant clusters.
• Use a multi-branch architecture with group normalization to improve training stability and feature representation across scales.
• Implement a hierarchical fusion module to combine multi-level self-attention features, enabling effective integration of local and global context.
• Leverage group normalization throughout the network to improve optimization and generalization during training.
• Train the model end-to-end using density map regression with L1 and L2 loss, minimizing MAE and MSE for count estimation.

Experimental results

Research questions

• RQ1 How can a deep learning model effectively handle large-scale variation in crowd counting without relying on fixed kernel sizes?
• RQ2 To what extent can long-range contextual modeling improve detection of isolated small clusters in crowded scenes?
• RQ3 Can a scale-adaptive attention mechanism enhance performance on both large and small objects across diverse scales?
• RQ4 How does hierarchical fusion of multi-level self-attention features improve overall counting accuracy?
• RQ5 Does SACANet generalize better than state-of-the-art methods on benchmarks with extreme scale variation and scattered clusters?

Key findings

• SACANet achieves a MAE of 64.4 on the ShanghaiTech A dataset and 7.8 on ShanghaiTech B, outperforming the previous SOTA method SANet (MAE 67.0 and 8.4 respectively).
• On the VisDrone2019 Vehicle dataset, SACANet achieves a MAE of 11.3, significantly lower than MCNN (13.1), demonstrating superior performance in extremely crowded scenes.
• The model reduces MAE by over 7% on ShanghaiTech A and about 4% on ShanghaiTech B compared to SANet, indicating strong gains in accuracy.
• Ablation studies confirm that each component—pyramid context module, scale-adaptive self-attention, and hierarchical fusion—contributes to performance gains, with the full model achieving the best results.
• SACANet consistently outperforms MCNN across all scale variation (CV) and isolated cluster (DVI) subsets, with lower MAE and MSE, especially in high-CV and high-DVI scenarios.
• The use of group normalization improves training optimality, contributing to better convergence and generalization across datasets.