[Paper Review] Scale-Aware Attention Network for Crowd Counting

Rahul Rama Varior, Bing Shuai+2 more|arXiv (Cornell University)|Jan 17, 2019

Video Surveillance and Tracking MethodsComputer Science29 references20 citations

TL;DR

This paper proposes a scale-aware attention network that generates multi-scale density predictions from different layers of a CNN in a single forward pass, using a learnable soft attention mechanism and a scale-aware loss to improve crowd counting. It achieves state-of-the-art performance, with a 25% relative improvement on UCF-QNRF and ~10% gains on other datasets, even without additional bells and whistles.

ABSTRACT

In crowd counting datasets, people appear at different scales, depending on their distance to the camera. To address this issue, we propose a novel multi-branch scale-aware attention network that exploits the hierarchical structure of convolutional neural networks and generates, in a single forward pass, multi-scale density predictions from different layers of the architecture. To aggregate these maps into our final prediction, we present a new soft attention mechanism that learns a set of gating masks. Furthermore, we introduce a scale-aware loss function to regularize the training of different branches and guide them to specialize on a particular scale. As this new training requires ground-truth annotations for the size of each head, we also propose a simple, yet effective technique to estimate it automatically. Finally, we present an ablation study on each of these components and compare our approach against the literature on 4 crowd counting datasets: UCF-QNRF, ShanghaiTech A & B and UCF_CC_50. Without bells and whistles, our approach achieves state-of-the-art on all these datasets. We observe a remarkable improvement on the UCF-QNRF (25%) and a significant one on the others (around 10%).

Motivation & Objective

• Address the challenge of variable person scale in crowd counting due to varying distances from the camera.
• Improve accuracy in dense crowd scenarios where people appear at vastly different scales.
• Develop a method that leverages hierarchical features from multiple network layers for scale-aware predictions.
• Introduce a learnable attention mechanism to dynamically weight multi-scale predictions.
• Design a scale-aware loss function to guide each network branch to specialize in a specific scale range.

Proposed method

• Proposes a multi-branch architecture that extracts density maps from different layers of a CNN, each sensitive to a specific scale range.
• Introduces a soft attention mechanism that learns gating masks to adaptively aggregate multi-scale density maps into a final prediction.
• Designs a scale-aware loss function that regularizes each branch to specialize in a particular scale by using scale-specific supervision.
• Develops an automatic head size estimation technique to generate pseudo-scale annotations when ground-truth sizes are unavailable.
• Employs a single forward pass to produce all multi-scale predictions and attention-based fusion, ensuring inference efficiency.
• Uses feature maps from multiple stages of the backbone network to capture both fine-grained and coarse-scale crowd patterns.

Experimental results

Research questions

• RQ1 Can a single deep neural network effectively predict crowd density across multiple scales using hierarchical features?
• RQ2 How can attention mechanisms be designed to optimally fuse multi-scale density maps without requiring additional supervision?
• RQ3 To what extent does a scale-aware loss function improve the specialization of individual branches for specific scale ranges?
• RQ4 How effective is the proposed automatic head size estimation technique in enabling scale-aware training without manual scale annotations?
• RQ5 Does the integration of attention and scale-aware loss lead to consistent performance gains across diverse crowd counting benchmarks?

Key findings

• The proposed method achieves state-of-the-art performance on all four benchmark datasets: UCF-QNRF, ShanghaiTech A & B, and UCF_CC_50.
• On UCF-QNRF, the model achieves a 25% relative improvement over prior methods, demonstrating exceptional gains in extreme density scenarios.
• On ShanghaiTech A and B, and UCF_CC_50, the method achieves significant improvements of approximately 10% over previous state-of-the-art approaches.
• The ablation study confirms that both the scale-aware loss and the soft attention mechanism are critical for performance gains.
• The automatic head size estimation technique enables effective scale supervision without requiring manual scale annotations.
• The model achieves these results with a single forward pass, maintaining high inference efficiency despite multi-scale processing.