[Paper Review] Crowd Counting with Deep Structured Scale Integration Network

Lingbo Liu, Zhilin Qiu+4 more|arXiv (Cornell University)|Aug 23, 2019

Video Surveillance and Tracking Methods45 references36 citations

TL;DR

DSSINet introduces a CRF-based Structured Feature Enhancement Module to mutually refine multiscale crowd features and a Dilated Multiscale Structural Similarity loss to enforce local scale consistency, achieving state-of-the-art results on multiple benchmarks.

ABSTRACT

Automatic estimation of the number of people in unconstrained crowded scenes is a challenging task and one major difficulty stems from the huge scale variation of people. In this paper, we propose a novel Deep Structured Scale Integration Network (DSSINet) for crowd counting, which addresses the scale variation of people by using structured feature representation learning and hierarchically structured loss function optimization. Unlike conventional methods which directly fuse multiple features with weighted average or concatenation, we first introduce a Structured Feature Enhancement Module based on conditional random fields (CRFs) to refine multiscale features mutually with a message passing mechanism. In this module, each scale-specific feature is considered as a continuous random variable and passes complementary information to refine the features at other scales. Second, we utilize a Dilated Multiscale Structural Similarity loss to enforce our DSSINet to learn the local correlation of people's scales within regions of various size, thus yielding high-quality density maps. Extensive experiments on four challenging benchmarks well demonstrate the effectiveness of our method. Specifically, our DSSINet achieves improvements of 9.5% error reduction on Shanghaitech dataset and 24.9% on UCF-QNRF dataset against the state-of-the-art methods.

Motivation & Objective

• Address severe scale variation in crowd scenes by learning robust multiscale feature representations.
• Leverage structured information sharing across scales via a CRF-based feature refinement mechanism.
• Impose a loss that captures local scale correlations across regions of varying sizes using a dilated MS-SSIM formulation.
• Produce high-quality density maps through a top-down fusion of side outputs with shared-parameter subnetworks.

Proposed method

• Use three parallel subnetworks with shared parameters processing different scaled versions of the same image.
• Introduce a Structured Feature Enhancement Module (SFEM) based on conditional random fields to mutually refine multiscale features via a message-passing scheme.
• Generate multiple side-output density maps from refined features and fuse them top-down to obtain a high-resolution density map.
• Propose a Dilated Multiscale Structural Similarity (DMS-SSIM) loss implemented with a fixed Gaussian kernel and dilated convolutions to measure SSIM over regions of varying sizes.
• Optimize with DMS-SSIM loss to enforce local scale correlation and density map consistency across scales.

Experimental results

Research questions

• RQ1 How can multiscale features be refined to better handle extreme scale variation in crowded scenes?
• RQ2 Can a CRF-based mutual refinement between scale-specific features improve robustness to scale changes?
• RQ3 Does a dilated MS-SSIM loss better capture local scale correlations than traditional losses in crowd counting?
• RQ4 What is the effectiveness and efficiency of the proposed DSSINet architecture across standard crowd counting benchmarks?

Key findings

• DSSINet achieves state-of-the-art performance on multiple benchmarks, notably 9.5% MAE reduction on Shanghaitech Part A and 24.9% MAE reduction on UCF-QNRF over prior methods (relative figures from the paper).
• SFEM (CRF-based feature refinement) significantly improves multiscale feature robustness compared with simple fusion strategies.
• DMS-SSIM loss with dilations (m=5 shown as best) yields the best MAE/MSE, outperforming Euclidean and SSIM-based losses.
• The model uses 8.858 million parameters and runs at about 450 ms per 720x576 frame on a 1080 GPU, with the backbone accounting for the majority of parameters, offering favorable accuracy/complexity trade-offs.