[Paper Review] Learning Independent Instance Maps for Crowd Localization

Junyu Gao, Tao Han+3 more|arXiv (Cornell University)|Dec 8, 2020

Video Surveillance and Tracking MethodsComputer Science59 references33 citations

TL;DR

The paper introduces Independent Instance Maps segmentation (IIM) with a differentiable binarization module to localize individual heads in crowds, achieving state-of-the-art results on NWPU-Crowd Localization and strong performance across multiple datasets.

ABSTRACT

Accurately locating each head's position in the crowd scenes is a crucial task in the field of crowd analysis. However, traditional density-based methods only predict coarse prediction, and segmentation/detection-based methods cannot handle extremely dense scenes and large-range scale-variations crowds. To this end, we propose an end-to-end and straightforward framework for crowd localization, named Independent Instance Map segmentation (IIM). Different from density maps and boxes regression, each instance in IIM is non-overlapped. By segmenting crowds into independent connected components, the positions and the crowd counts (the centers and the number of components, respectively) are obtained. Furthermore, to improve the segmentation quality for different density regions, we present a differentiable Binarization Module (BM) to output structured instance maps. BM brings two advantages into localization models: 1) adaptively learn a threshold map for different images to detect each instance more accurately; 2) directly train the model using loss on binary predictions and labels. Extensive experiments verify the proposed method is effective and outperforms the-state-of-the-art methods on the five popular crowd datasets. Significantly, IIM improves F1-measure by 10.4% on the NWPU-Crowd Localization task. The source code and pre-trained models will be released at https://github.com/taohan10200/IIM.

Motivation & Objective

• Motivate accurate head localization in extremely dense crowds beyond density or box-based methods.
• Propose Independent Instance Maps (IIM) where each instance is non-overlapped and extractable via connected components.
• Introduce a differentiable Binarization Module (BM) to produce structured instance maps.
• Incorporate a Pixel-level Binarization Module to adapt thresholds per pixel region for scale-variation robustness.
• Demonstrate superior localization and competitive counting across standard crowd datasets.

Proposed method

• Represent crowd regions as confidence maps and segment them into independent connected components to obtain head centers and counts.
• Introduce a differentiable binarization layer to convert confidence maps into binary instance maps without requiring extra supervision.
• Embed a threshold encoder to produce image-level or pixel-level thresholds guiding binarization.
• Use a Pixel-level Binarization Module (PBM) to generate pixel-wise thresholds that adapt to scale variation and spatial distribution.
• Train with a combination of regression losses for confidence maps and L1 losses for threshold maps, with gradient flow controlled to balance backpropagation between components.
• Output localization by detecting 4-connected components and extracting centers within independent instances.

Experimental results

Research questions

• RQ1 Can independent, non-overlapping instance maps improve localization accuracy in highly dense crowds compared to density- or detection-based approaches?
• RQ2 Does a differentiable binarization layer enable end-to-end optimization and better edge segmentation for tiny or occluded heads?
• RQ3 Do image-level and pixel-level threshold learning strategies improve localization and counting under large-range scale variations?
• RQ4 Can a threshold encoder integrated with a confidence predictor adapt to varying crowd densities to improve localization robustness?

Key findings

• IIM achieves state-of-the-art localization on NWPU-Crowd, ranking first on the Localization benchmark with F1-m = 76.2% and MAE = 87.1 on the test set (per Table II).
• The method improves F1-measure by about 9.0% over prior approaches on NWPU-Crowd Localization.
• Pixel-level threshold learning (PBM) provides more precise localization than image-level thresholding (IBM).
• IIM achieves strong performance across multiple datasets, showing robustness to negative samples and dense crowds.
• The approach yields higher precision and competitive recall, outperforming several detection- and density-based methods on ShanghaiTech Part A/B, UCF-QNRF, and FDST in localization tasks (as summarized in Tables IV and related results).
• Ablation studies demonstrate significant gains from IBM/PBM over fixed thresholds and show the beneficial impact of incorporating L1 loss and gradient flow from the localization objective.