[Paper Review] FusionCount: Efficient Crowd Counting via Multiscale Feature Fusion

Yiming Ma, Víctor Sánchez+1 more|arXiv (Cornell University)|Feb 28, 2022

Video Surveillance and Tracking Methods7 citations

TL;DR

FusionCount proposes an efficient crowd counting model that leverages multiscale feature fusion from multiple encoder layers instead of relying on additional multiscale modules, significantly reducing computational cost while achieving state-of-the-art performance on ShanghaiTech A and B benchmarks with lower MAE and RMSE.

ABSTRACT

State-of-the-art crowd counting models follow an encoder-decoder approach. Images are first processed by the encoder to extract features. Then, to account for perspective distortion, the highest-level feature map is fed to extra components to extract multiscale features, which are the input to the decoder to generate crowd densities. However, in these methods, features extracted at earlier stages during encoding are underutilised, and the multiscale modules can only capture a limited range of receptive fields, albeit with considerable computational cost. This paper proposes a novel crowd counting architecture (FusionCount), which exploits the adaptive fusion of a large majority of encoded features instead of relying on additional extraction components to obtain multiscale features. Thus, it can cover a more extensive scope of receptive field sizes and lower the computational cost. We also introduce a new channel reduction block, which can extract saliency information during decoding and further enhance the model's performance. Experiments on two benchmark databases demonstrate that our model achieves state-of-the-art results with reduced computational complexity.

Motivation & Objective

• Address the limitation of existing crowd counting models that underutilize shallow encoder features and rely on costly multiscale modules.
• Reduce computational complexity in multiscale feature extraction while maintaining or improving accuracy.
• Enhance model performance by introducing a novel channel reduction block that extracts saliency during decoding.
• Enable comprehensive receptive field coverage (6 to 192) using features from multiple encoder stages without additional modules.
• Achieve state-of-the-art performance with lower floating-point operations compared to existing VGG-based models.

Proposed method

• Utilizes the first 17 layers of VGG-16 as the encoder, collecting feature maps from multiple stages (starting from layer 3) to capture diverse receptive field sizes.
• Applies first-phase feature fusion by grouping features of the same spatial resolution and fusing them using learnable weights to generate multiscale representations.
• Introduces a contrast feature-based fusion mechanism that computes attention weights from contrast features rather than raw features, improving fusion quality.
• Employs a channel reduction module with point-wise convolution and chained dilated convolutions to reduce channel dimensions while preserving salient information.
• Uses a decoder that fuses the multiscale features from the first-phase fusion and the channel-reduced features to generate the final density map.
• Leverages standard loss functions (MSE, MAE) for training, with no additional architectural innovations beyond the fusion and reduction modules.

Experimental results

Research questions

• RQ1 Can multiscale feature representation be effectively captured without additional modules by leveraging features from multiple encoder layers?
• RQ2 How does fusing features from multiple encoder stages compare to using only the highest-level feature map in terms of accuracy and efficiency?
• RQ3 Does the proposed contrast feature-based fusion mechanism improve performance over direct feature fusion?
• RQ4 To what extent does the channel reduction module enhance model performance by focusing on salient features during decoding?
• RQ5 Can a model achieve state-of-the-art crowd counting accuracy with significantly reduced computational cost compared to existing SOTA methods?

Key findings

• On the ShanghaiTech B dataset, FusionCount achieves the lowest MAE (6.9) and RMSE (11.8), outperforming all compared models including CSRNet, CAN, BL, and DM-Count.
• On ShanghaiTech A, FusionCount achieves the second-lowest MAE (13.4), demonstrating strong performance despite the dataset's training difficulty.
• The variant model using raw features instead of contrast features achieves higher MAE (7.6) and RMSE (12.9), confirming the effectiveness of the contrast feature fusion mechanism.
• Ablation studies show that both the point-wise convolution and the dilated convolutional layers in the channel reduction module are essential, as removing either increases MAE and RMSE.
• FusionCount achieves state-of-the-art results with significantly reduced computational complexity, as evidenced by lower FLOPs compared to models like CAN and CSRNet.
• Qualitative results show accurate density estimation even in highly crowded scenes (e.g., 1,600+ people with 1.98% relative error) and under strong scale variation, confirming robustness.