[Paper Review] Crowd Scene Analysis by Output Encoding

Yao Xue, Siming Liu+2 more|arXiv (Cornell University)|Jan 27, 2020

Anomaly Detection Techniques and Applications49 references4 citations

TL;DR

This paper proposes a Compressed Sensing-based Output Encoding (CSOE) framework for crowd scene analysis that treats crowd localization as signal regression in a compressed encoding space, significantly improving localization accuracy in highly crowded scenes. By integrating Multiple Dilated Convolution Branches (MDCB) and Adaptive Receptive Field Weighting (ARFW), the method effectively handles large scale variations and achieves state-of-the-art performance on four benchmark datasets, especially excelling in dense crowd scenarios.

ABSTRACT

Crowd scene analysis receives growing attention due to its wide applications. Grasping the accurate crowd location (rather than merely crowd count) is important for spatially identifying high-risk regions in congested scenes. In this paper, we propose a Compressed Sensing based Output Encoding (CSOE) scheme, which casts detecting pixel coordinates of small objects into a task of signal regression in encoding signal space. CSOE helps to boost localization performance in circumstances where targets are highly crowded without huge scale variation. In addition, proper receptive field sizes are crucial for crowd analysis due to human size variations. We create Multiple Dilated Convolution Branches (MDCB) that offers a set of different receptive field sizes, to improve localization accuracy when objects sizes change drastically in an image. Also, we develop an Adaptive Receptive Field Weighting (ARFW) module, which further deals with scale variation issue by adaptively emphasizing informative channels that have proper receptive field size. Experiments demonstrate the effectiveness of the proposed method, which achieves state-of-the-art performance across four mainstream datasets, especially achieves excellent results in highly crowded scenes. More importantly, experiments support our insights that it is crucial to tackle target size variation issue in crowd analysis task, and casting crowd localization as regression in encoding signal space is quite effective for crowd analysis.

Motivation & Objective

• To address the challenge of accurate crowd localization in highly congested scenes where density-based methods fail to localize individual objects.
• To overcome the limitations of direct pixel coordinate regression, which suffers from position shift and size scaling errors in dense crowds.
• To improve robustness to large scale variations in human head sizes across a single image.
• To develop a method that maintains high localization precision even when objects are densely packed and occluded.
• To validate that casting localization as regression in an encoded signal space is more effective than direct coordinate prediction for crowd scenes.

Proposed method

• The CSOE module encodes sparse crowd locations (head centroids) into a compressed, dense signal vector, transforming the localization task into a signal regression problem.
• A CNN regressor predicts the compressed signal vector, which is then reconstructed into sparse pixel coordinates using a reconstruction algorithm.
• Multiple Dilated Convolution Branches (MDCB) are introduced to provide multiple, fixed receptive field sizes, enhancing detection across varying object scales.
• Adaptive Receptive Field Weighting (ARFW) dynamically emphasizes feature channels with receptive fields most suitable for object scale, improving generalization.
• The model is trained end-to-end with a sparse reconstruction loss, preserving spatial relationships through compressed sensing principles.
• Center pooling is applied to enhance feature learning within object centers, improving localization precision.

Experimental results

Research questions

• RQ1 Can compressing the output space via Compressed Sensing improve localization accuracy in highly crowded scenes compared to direct coordinate regression?
• RQ2 How effective is the use of multiple dilated convolution branches in handling large variations in human head size within a single image?
• RQ3 Does adaptive weighting of receptive fields based on scale improve localization performance in complex crowd scenes?
• RQ4 To what extent does the proposed method outperform existing state-of-the-art approaches in both counting and localization tasks across diverse datasets?
• RQ5 Is the combination of CSOE, MDCB, and ARFW synergistic in enhancing performance, particularly in high-density scenarios?

Key findings

• The complete model (CSOE + MDCB + CP + ARFW) achieves the highest F1-score of 0.831 on the ShanghaiTech (Sh-A) dataset, outperforming all ablation configurations.
• The model achieves an F1-score of 0.834 on the ShanghaiTech (Sh-B) dataset, demonstrating strong generalization across different crowd densities.
• The ARFW module, when combined with MDCB, improves F1-score by 0.027–0.045 points over configurations without adaptive weighting, confirming its effectiveness in scale handling.
• The MDCB+ARFW configuration achieves the highest F1-scores among all two-component configurations, indicating that scale variation is a critical factor in crowd localization.
• The ablation study shows that CSOE alone achieves the best performance among single-component configurations, supporting the core insight that signal-space regression is more effective than pixel-space regression.
• The model achieves state-of-the-art performance on all four benchmark datasets—ShanghaiTech, UCF, and WorldExpo—especially in high-density scenes where traditional methods fail.