[Paper Review] TransCrowd: weakly-supervised crowd counting with transformers

Dingkang Liang, Xiwu Chen+3 more|arXiv (Cornell University)|Apr 19, 2021

Video Surveillance and Tracking MethodsComputer Science41 references25 citations

TL;DR

TransCrowd introduces a pure Transformer approach for weakly supervised crowd counting, reformulating image-to-count as sequence-to-count and achieving state-of-the-art results among count-level methods. It compares two regression heads and demonstrates faster convergence with global average pooling.

ABSTRACT

The mainstream crowd counting methods usually utilize the convolution neural network (CNN) to regress a density map, requiring point-level annotations. However, annotating each person with a point is an expensive and laborious process. During the testing phase, the point-level annotations are not considered to evaluate the counting accuracy, which means the point-level annotations are redundant. Hence, it is desirable to develop weakly-supervised counting methods that just rely on count-level annotations, a more economical way of labeling. Current weakly-supervised counting methods adopt the CNN to regress a total count of the crowd by an image-to-count paradigm. However, having limited receptive fields for context modeling is an intrinsic limitation of these weakly-supervised CNN-based methods. These methods thus cannot achieve satisfactory performance, with limited applications in the real world. The transformer is a popular sequence-to-sequence prediction model in natural language processing (NLP), which contains a global receptive field. In this paper, we propose TransCrowd, which reformulates the weakly-supervised crowd counting problem from the perspective of sequence-to-count based on transformers. We observe that the proposed TransCrowd can effectively extract the semantic crowd information by using the self-attention mechanism of transformer. To the best of our knowledge, this is the first work to adopt a pure transformer for crowd counting research. Experiments on five benchmark datasets demonstrate that the proposed TransCrowd achieves superior performance compared with all the weakly-supervised CNN-based counting methods and gains highly competitive counting performance compared with some popular fully-supervised counting methods.

Motivation & Objective

• Motivate the development of count-level (weakly-supervised) crowd counting to reduce annotation effort beyond point-level density maps.
• Leverage Transformer global receptive fields to capture long-range crowd context for counting.
• Propose two Transformer-based counting architectures (TransCrowd-Token and TransCrowd-GAP) and compare their effectiveness.
• Demonstrate that pure Transformer models can achieve competitive or superior counting accuracy compared to fully-supervised CNN-based methods on standard datasets.

Proposed method

• Transform the input image into a sequence of fixed-size patches and embed them with positional information.
• Apply a Transformer-encoder (12 layers; multi-head self-attention with residual connections) to obtain a rich global representation of the image patches.
• Introduce two regression head designs: TransCrowd-Token uses a learnable regression token; TransCrowd-GAP uses global average pooling of visual tokens before regression.
• Train with L1 loss to predict the total crowd count per image.
• Pre-train on ImageNet and fine-tune on crowd counting datasets; resize images and use standard data augmentation for training.

Experimental results

Research questions

• RQ1 Can a pure Transformer-based network, trained with count-level supervision, achieve competitive crowd counting performance without point-level density supervision?
• RQ2 Do regression head designs (regression token vs. global pooled tokens) influence counting accuracy and convergence speed?
• RQ3 How does TransCrowd compare to existing weakly-supervised and fully-supervised methods on standard benchmarks across varying crowd densities?
• RQ4 What qualitative differences in attention maps emerge between the two regression head variants, and how do they relate to counting accuracy?

Key findings

• TransCrowd-GAP achieves higher counting accuracy and faster convergence than TransCrowd-Token across multiple datasets.
• TransCrowd substantially outperforms existing weakly-supervised CNN-based methods and is highly competitive with fully-supervised methods.
• On JHU-Crowd++ testing set, TransCrowd-GAP improves CSRNet by notable margins (MAE and MSE) and even surpasses some fully-supervised methods on certain datasets.
• Attention visualizations show TransCrowd-GAP producing more reasonable attention maps than TransCrowd-Token, which helps reduce counting error.
• The method demonstrates strong performance on NWPU-Crowd and JHU-Crowd++ with large-scale datasets, likely due to the Transformer’s global receptive field.