[Paper Review] CCTrans: Simplifying and Improving Crowd Counting with Transformer

Ye Tian, Xiangxiang Chu+1 more|arXiv (Cornell University)|Sep 29, 2021

Video Surveillance and Tracking MethodsComputer Science48 references64 citations

TL;DR

CCTrans uses a pyramid vision transformer backbone with pyramid feature aggregation and a lightweight multi-scale dilated regression head to achieve state-of-the-art crowd counting under both fully- and weakly-supervised settings. It ranks No.1 on NWPU-Crowd and outperforms prior methods on several benchmarks.

ABSTRACT

Most recent methods used for crowd counting are based on the convolutional neural network (CNN), which has a strong ability to extract local features. But CNN inherently fails in modeling the global context due to the limited receptive fields. However, the transformer can model the global context easily. In this paper, we propose a simple approach called CCTrans to simplify the design pipeline. Specifically, we utilize a pyramid vision transformer backbone to capture the global crowd information, a pyramid feature aggregation (PFA) model to combine low-level and high-level features, an efficient regression head with multi-scale dilated convolution (MDC) to predict density maps. Besides, we tailor the loss functions for our pipeline. Without bells and whistles, extensive experiments demonstrate that our method achieves new state-of-the-art results on several benchmarks both in weakly and fully-supervised crowd counting. Moreover, we currently rank No.1 on the leaderboard of NWPU-Crowd. Our code will be made available.

Motivation & Objective

• Motivate improving crowd counting by better modeling global context beyond limited CNN receptive fields.
• Propose a simplified, transformer-based pipeline to unify fully- and weakly-supervised crowd counting.
• Leverage global context with a pyramid transformer and aggregate multi-level features for accurate density maps.
• Develop a lightweight regression head with multi-scale receptive fields and tailored losses.
• Demonstrate state-of-the-art performance on five standard benchmarks.

Proposed method

• Transform image to a 1D sequence of patches and process with a pyramid transformer backbone (Twins) to capture global context.
• Reshape stage outputs to 2D feature maps and apply Pyramid Feature Aggregation to combine low- and high-level information.
• Use a regression head with Multi-scale Dilated Convolution to regress the density map with multi-scale receptive fields.
• Train with tailored losses: a fully-supervised density regression loss combining L1, OT, and smoothed L2 terms; and a weakly-supervised smooth L1 counting loss.
• Optionally operate under weak supervision by summing density map pixels to obtain crowd counts.
• Incorporate two design blocks (PFA and MDC) and a careful loss setup to achieve strong performance without excessive complexity.

Experimental results

Research questions

• RQ1 Can a transformer-based backbone with simple feature aggregation match or exceed CNN-based crowd counting pipelines across standard benchmarks under both supervision modes?
• RQ2 Does a lightweight multi-scale dilated regression head suffice for accurate density map regression when global context is well-modeled by a pyramid transformer?
• RQ3 What is the impact of tailored loss functions on fully- and weakly-supervised crowd counting performance?
• RQ4 How does the proposed CCTrans compare to transformer-based counterpoints like TransCrowd and BCCT across datasets?

Key findings

• CCTrans achieves new state-of-the-art results on multiple popular crowd counting benchmarks under both fully- and weakly-supervised settings.
• On NWPU-Crowd, CCTrans ranks No.1 on the leaderboard (validation and test performance reported).
• A simple regression head with multi-scale dilated convolution (MDC) effectively leverages multi-scale context for density regression.
• Pyramid Feature Aggregation (PFA) substantially improves performance by fusing multi-stage transformer features.
• A carefully designed loss (smooth L1 for weakly-supervised; L1+OT+smoothed L2 for fully-supervised) improves robustness and accuracy.
• Ablation studies confirm the contributions of PFA and MDC to the final performance.