[Paper Review] Weighing Counts: Sequential Crowd Counting by Reinforcement Learning

Liang Liu, Hao Lü+4 more|arXiv (Cornell University)|Jul 16, 2020

Video Surveillance and Tracking Methods50 references4 citations

TL;DR

This paper proposes LibraNet, a novel crowd counting method that formulates counting as a sequential decision problem inspired by scale weighing, using deep reinforcement learning to iteratively select weights (actions) to approximate the true crowd count. It achieves state-of-the-art performance on ShanghaiTech, UCF_CC_50, and UCF-QNRF benchmarks by modeling the count estimation process as a Deep Q-Network (DQN)-based balancing task, where the agent learns to minimize error through feedback from a simulated scale's needle (Q-values).

ABSTRACT

We formulate counting as a sequential decision problem and present a novel crowd counting model solvable by deep reinforcement learning. In contrast to existing counting models that directly output count values, we divide one-step estimation into a sequence of much easier and more tractable sub-decision problems. Such sequential decision nature corresponds exactly to a physical process in reality scale weighing. Inspired by scale weighing, we propose a novel 'counting scale' termed LibraNet where the count value is analogized by weight. By virtually placing a crowd image on one side of a scale, LibraNet (agent) sequentially learns to place appropriate weights on the other side to match the crowd count. At each step, LibraNet chooses one weight (action) from the weight box (the pre-defined action pool) according to the current crowd image features and weights placed on the scale pan (state). LibraNet is required to learn to balance the scale according to the feedback of the needle (Q values). We show that LibraNet exactly implements scale weighing by visualizing the decision process how LibraNet chooses actions. Extensive experiments demonstrate the effectiveness of our design choices and report state-of-the-art results on a few crowd counting benchmarks. We also demonstrate good cross-dataset generalization of LibraNet. Code and models are made available at: https://git.io/libranet

Motivation & Objective

• To address the limitations of direct regression-based crowd counting models that struggle with dense, occluded, and varying-sized crowds.
• To model crowd counting as a sequential decision-making process, inspired by human counting behavior and physical scale weighing.
• To improve accuracy and generalization by decomposing the global count estimation into a series of tractable sub-decisions using reinforcement learning.
• To enable plug-in integration with existing local count estimation models for enhanced performance.
• To demonstrate cross-dataset generalization and robustness to challenging visual conditions such as blur and illumination variation.

Proposed method

• Formulates crowd counting as a sequential decision problem where the agent (LibraNet) virtually places weights on a scale to match the true crowd count, analogous to physical scale weighing.
• Uses a Deep Q-Network (DQN) to learn optimal weight selection policies based on image features and the current state of the scale (i.e., sum of previously placed weights).
• Defines a pre-specified action pool of discrete weight values (e.g., ±0.01, ±0.02, ..., ±5) from which the agent selects actions at each step.
• Employs a reward function based on the deviation between the current total weight and the ground-truth count, with feedback from the needle's position (Q-value) guiding learning.
• Introduces a two-stage pipeline: first, a local count model (e.g., TasselNetv2) generates patch-level counts; second, LibraNet aggregates these via sequential action selection to produce the final global count.
• Applies inverse-quantization to recover continuous count values from discrete interval predictions, enabling regression-based adaptation of the method.

Experimental results

Research questions

• RQ1 Can crowd counting be effectively modeled as a sequential decision-making process that mimics human counting behavior?
• RQ2 Does formulating count estimation as scale weighing—using reinforcement learning to balance a virtual scale—improve accuracy and robustness over direct regression?
• RQ3 How does LibraNet perform compared to state-of-the-art models on standard crowd counting benchmarks such as ShanghaiTech, UCF_CC_50, and UCF-QNRF?
• RQ4 To what extent can LibraNet generalize across datasets, especially under distribution shifts like long-tailed data or varying image quality?
• RQ5 Can the proposed method be effectively integrated as a plug-in to existing local count estimation models to boost performance?

Key findings

• LibraNet achieves state-of-the-art performance on the ShanghaiTech Part_A, Part_B, UCF_CC_50, and UCF-QNRF benchmarks, outperforming existing methods including TasselNetv2 and other SOTA models.
• The method demonstrates strong cross-dataset generalization, generalizing well from ShanghaiTech to UCF-QNRF and UCF_CC_50 without fine-tuning.
• Ablation studies confirm that the design of the action pool and reward function significantly improves performance, with the full LibraNet setup yielding the best results.
• Visualization of the decision process confirms that LibraNet effectively implements scale weighing: it starts with large weight adjustments and refines with smaller weights as it approaches the true count.
• The integration of LibraNet as a plug-in to TasselNetv2 significantly improves its performance, proving the method’s compatibility and effectiveness.
• Failure cases reveal limitations under extreme conditions such as long-tailed data distributions (causing underestimation), blur, and poor illumination, indicating areas for future improvement.