[Paper Review] Dual Path Multi-Scale Fusion Networks with Attention for Crowd Counting

Liang Zhu, Zhijian Zhao+4 more|arXiv (Cornell University)|Feb 4, 2019

Video Surveillance and Tracking MethodsComputer Science28 references69 citations

TL;DR

SFANet introduces a dual-path, multi-scale fusion network with an attention mechanism to produce high-resolution density maps and accurate crowd counts across diverse densities. It uses a VGG16-bn backbone with two fused paths (density map path and attention map path) trained end-to-end.

ABSTRACT

The task of crowd counting in varying density scenes is an extremely difficult challenge due to large scale variations. In this paper, we propose a novel dual path multi-scale fusion network architecture with attention mechanism named SFANet that can perform accurate count estimation as well as present high-resolution density maps for highly congested crowd scenes. The proposed SFANet contains two main components: a VGG backbone convolutional neural network (CNN) as the front-end feature map extractor and a dual path multi-scale fusion networks as the back-end to generate density map. These dual path multi-scale fusion networks have the same structure, one path is responsible for generating attention map by highlighting crowd regions in images, the other path is responsible for fusing multi-scale features as well as attention map to generate the final high-quality high-resolution density maps. SFANet can be easily trained in an end-to-end way by dual path joint training. We have evaluated our method on four crowd counting datasets (ShanghaiTech, UCF CC 50, UCSD and UCF-QRNF). The results demonstrate that with attention mechanism and multi-scale feature fusion, the proposed SFANet achieves the best performance on all these datasets and generates better quality density maps compared with other state-of-the-art approaches.

Motivation & Objective

• Address large head scale variations and background noise in crowd counting.
• Leverage multi-scale feature fusion to generate high-resolution density maps.
• Incorporate an attention pathway to highlight head regions and suppress background.
• Propose a multi-task loss combining Euclidean density loss and attention guidance.
• Demonstrate superior performance on standard crowd counting benchmarks.

Proposed method

• Use VGG16-bn backbone to extract multi-scale features (conv2-2, conv3-3, conv4-3, conv5-3).
• Construct a density map path (DMP) via feature pyramid fusion to produce high-resolution density maps.
• Construct an attention map path (AMP) with the same structure to learn head-region probabilities.
• Fuse DMP features with an attention map via element-wise multiplication to refine density features.
• Train with a multi-task loss: L = L_density + alpha * L_attention (alpha = 0.1).
• Generate ground-truth density maps by Gaussian-blurring head annotations; derive attention-ground-truth from density maps.

Experimental results

Research questions

• RQ1 Can a dual-path, multi-scale fusion network improve robustness to scale variation and background noise in crowd counting?
• RQ2 Does integrating an attention map path improve localization of head regions and density map quality?
• RQ3 Does the proposed multi-task loss accelerate convergence and boost counting accuracy?

Key findings

• SFANet achieves state-of-the-art or competitive MAE/MSE across ShanghaiTech, UCF_CC_50, UCF-QRNF, and UCSD datasets.
• On ShanghaiTech Part A, SFANet attains 59.8 MAE and 99.3 MSE; on Part B, 6.9 MAE and 10.9 MSE (per Table 1).
• On UCF_CC_50, SFANet achieves 219.6 MAE and 316.2 MSE.
• On UCF-QRNF, SFANet attains 100.8 MAE and 174.5 MSE.
• On UCSD, SFANet achieves 0.82 MAE and 1.07 MSE (lower is better).
• Ablation shows the attention path improves performance beyond VGG-DMP baselines, confirming its contribution.