[Paper Review] Point-Query Quadtree for Crowd Counting, Localization, and More

Chengxin Liu, Hao Lu+2 more|arXiv (Cornell University)|Aug 26, 2023

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes Point-Query Quadtree (PET), a novel Transformer-based framework for crowd counting and localization that treats crowd estimation as a decomposable point querying process. By using a data-dependent quadtree to dynamically split queries in dense regions and progressive rectangle window attention for efficient inference, PET achieves state-of-the-art performance (49.34 MAE on ShanghaiTech PartA) while enabling unified modeling across multiple learning paradigms including fully-supervised counting, partial annotation learning, and point annotation refinement.

ABSTRACT

We show that crowd counting can be viewed as a decomposable point querying process. This formulation enables arbitrary points as input and jointly reasons whether the points are crowd and where they locate. The querying processing, however, raises an underlying problem on the number of necessary querying points. Too few imply underestimation; too many increase computational overhead. To address this dilemma, we introduce a decomposable structure, i.e., the point-query quadtree, and propose a new counting model, termed Point quEry Transformer (PET). PET implements decomposable point querying via data-dependent quadtree splitting, where each querying point could split into four new points when necessary, thus enabling dynamic processing of sparse and dense regions. Such a querying process yields an intuitive, universal modeling of crowd as both the input and output are interpretable and steerable. We demonstrate the applications of PET on a number of crowd-related tasks, including fully-supervised crowd counting and localization, partial annotation learning, and point annotation refinement, and also report state-of-the-art performance. For the first time, we show that a single counting model can address multiple crowd-related tasks across different learning paradigms. Code is available at https://github.com/cxliu0/PET.

Motivation & Objective

• To unify diverse crowd-related tasks—such as fully-supervised counting, localization, partial annotation learning, and point annotation refinement—under a single, interpretable framework.
• To address the challenge of fixed query count in point-based crowd modeling, where too few queries cause underestimation and too many increase computational cost.
• To enable dynamic, data-dependent processing of sparse and dense crowd regions through a hierarchical quadtree structure.
• To improve inference efficiency and representation learning by introducing progressive rectangle window attention that captures local context with reduced memory usage.
• To provide an intuitive, steerable, and interpretable modeling of crowd where each query point physically corresponds to a person or background.

Proposed method

• PET formulates crowd counting as a decomposable point querying process, where arbitrary input points are used to predict whether each is a person and its location.
• A point-query quadtree structure enables data-dependent splitting: each query point can split into four new points in dense regions, adapting dynamically to local density.
• The model uses a CNN backbone to extract image features, followed by a transformer encoder with progressive rectangle window attention to encode context within localized spatial windows.
• A transformer decoder processes the point queries in parallel, computing attention within local rectangle windows to reduce computational cost and improve efficiency.
• The final prediction head outputs per-query probabilities and bounding box coordinates for each point, classifying it as 'person' or 'background'.
• The progressive rectangle window attention mechanism is applied to both encoder and decoder, with rectangle-shaped windows outperforming square or global attention due to better alignment with perspective priors in crowd scenes.

Experimental results

Research questions

• RQ1 Can crowd counting be unified across multiple learning paradigms—such as fully-supervised, partial annotation, and point refinement—using a single, interpretable framework?
• RQ2 How can a point-based querying mechanism dynamically adapt to varying crowd densities without predefining the number of queries?
• RQ3 What is the impact of local, progressive attention mechanisms on inference efficiency and performance in high-resolution crowd scenes?
• RQ4 Can a quadtree-based structure effectively balance computational cost and accuracy in both sparse and dense crowd regions?
• RQ5 To what extent does the interpretability of point queries improve model transparency and task adaptability?

Key findings

• PET achieves a state-of-the-art mean absolute error (MAE) of 49.34 on the ShanghaiTech PartA benchmark, outperforming prior methods.
• The point-query quadtree structure reduces MAE by approximately 4 on ShanghaiTech PartA and 11 on UCF-QNRF compared to baseline point queries, due to better handling of sparse and dense regions.
• Progressive rectangle window attention significantly improves performance, with a 51.93 MAE when used in the decoder and 49.34 when used in both encoder and decoder, outperforming square or global attention.
• PET uses only 20.9M parameters and achieves inference time of 0.097s on a 1024×1024 input, making it more parameter-efficient than methods like P2PNet and CLTR.
• Visualization of attention maps confirms that the model learns to attend to similar crowd patterns within rectangle windows, indicating effective local context encoding.
• The model demonstrates strong generalization across learning paradigms, successfully handling fully-supervised counting, partial annotation learning, and point annotation refinement with a single architecture.