[Paper Review] Congested Crowd Instance Localization with Dilated Convolutional Swin Transformer

Junyu Gao, Maoguo Gong+1 more|arXiv (Cornell University)|Aug 2, 2021

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes a Dilated Convolutional Swin Transformer (DCST) for precise instance localization in highly congested crowd scenes, integrating dilated convolutions into the Swin Transformer backbone to enhance large-receptive-field contextual feature learning. The method achieves state-of-the-art performance with an F1-measure of 77.5% and MAE of 84.2 on benchmark datasets, significantly improving localization in occluded and blurred regions.

ABSTRACT

Crowd localization is a new computer vision task, evolved from crowd counting. Different from the latter, it provides more precise location information for each instance, not just counting numbers for the whole crowd scene, which brings greater challenges, especially in extremely congested crowd scenes. In this paper, we focus on how to achieve precise instance localization in high-density crowd scenes, and to alleviate the problem that the feature extraction ability of the traditional model is reduced due to the target occlusion, the image blur, etc. To this end, we propose a Dilated Convolutional Swin Transformer (DCST) for congested crowd scenes. Specifically, a window-based vision transformer is introduced into the crowd localization task, which effectively improves the capacity of representation learning. Then, the well-designed dilated convolutional module is inserted into some different stages of the transformer to enhance the large-range contextual information. Extensive experiments evidence the effectiveness of the proposed methods and achieve state-of-the-art performance on five popular datasets. Especially, the proposed model achieves F1-measure of 77.5\% and MAE of 84.2 in terms of localization and counting performance, respectively.

Motivation & Objective

• Address the challenge of precise instance-level localization in extremely congested crowd scenes where small objects, mutual occlusion, and image blur degrade performance.
• Overcome the limitations of traditional models in extracting robust features under severe occlusion and blur by enhancing contextual representation capacity.
• Improve the feature extraction capability of vision transformers in dense crowd scenarios through strategic integration of dilated convolutions.
• Achieve state-of-the-art performance on multiple benchmark datasets for crowd instance localization and counting.

Proposed method

• Integrate a window-based Swin Transformer as the backbone encoder to capture hierarchical, global-local representations in crowd scenes.
• Insert a novel Dilated Convolutional Block (DCB) with dilatation rates of 2 and 3 into Stage 3 and 4 of the Swin Transformer to expand the receptive field.
• Reorganize the transformer’s feature maps into a spatial format after each stage to enable effective application of dilated convolutions.
• Use a Feature Pyramid Network (FPN) decoder to generate a full-resolution segmentation map matching the input size for instance-level prediction.
• Design a flexible DCB module that preserves local details while encoding long-range contextual dependencies across multiple scales.
• Optimize the placement and dilatation rates of DCB through ablation studies to maximize performance on localization and counting metrics.

Figure 1: Two problems in some typical crowd scenes: Tiny Objects & Mutual Occlusion and Blurred Regions, which cause that the current crowd localization models perform poorly in dense crowd scenes.

Experimental results

Research questions

• RQ1 Where in the Swin Transformer architecture should dilated convolutional blocks be inserted to maximize localization performance in dense crowds?
• RQ2 What dilatation rate configuration in the DCB module yields the best trade-off between local detail preservation and long-range context encoding?
• RQ3 How does the threshold selection affect the trade-off between localization F1-measure and counting MAE in crowd instance localization?
• RQ4 To what extent does the proposed DCST improve performance over standard Swin Transformer and other state-of-the-art models on benchmark datasets?

Key findings

• The proposed DCST achieves a state-of-the-art F1-measure of 77.5% and MAE of 84.2 on the five benchmark datasets, demonstrating superior localization and counting performance.
• Inserting the DCB into Stage 3 and 4 of the Swin Transformer yields the best performance, with an F1-measure of 81.4% on the NWPU-Crowd validation set.
• The DCB with dilatation rates {2, 3} achieves the optimal balance, yielding 81.4% F1-measure, 84.1% precision, and 79.0% recall on the NWPU dataset.
• A threshold of 0.44 maximizes F1-measure (81.45%), while a threshold of 0.32 yields the lowest MAE (38.71), revealing a trade-off between localization and counting objectives.
• The model significantly outperforms standard Swin Transformer and other SOTA methods, especially in handling tiny, occluded, and blurred crowd regions.
• Ablation studies confirm that inserting DCB into shallow stages (1 and 2) harms local feature learning, while deeper insertion preserves structural details better.

Figure 2: The flowchart of the proposed Dilated Convolutional Swin Transformer and FPN decoder (DCST+FPN). Notably, In Stage 3 and 4, the Dilated Convolutional Block (DCB) is applied to the top of the Swin Transformer Block (STB). By a Feature Pyramid Networks (FPN) decoder, the features from differ