[Paper Review] Estimating People Flows to Better Count Them in Crowded Scenes

Weizhe Liu, Mathieu Salzmann+1 more|arXiv (Cornell University)|Nov 25, 2019

Video Surveillance and Tracking Methods61 references4 citations

TL;DR

This paper proposes estimating people flows between consecutive frames in crowded scenes to improve crowd counting accuracy, leveraging flow-based density estimation with strong conservation constraints. By regressing flows instead of densities and exploiting optical flow correlation, the method achieves state-of-the-art performance without deeper networks, reducing MAE by up to 21.5% on benchmark datasets.

ABSTRACT

Modern methods for counting people in crowded scenes rely on deep networks to estimate people densities in individual images. As such, only very few take advantage of temporal consistency in video sequences, and those that do only impose weak smoothness constraints across consecutive frames. In this paper, we advocate estimating people flows across image locations between consecutive images and inferring the people densities from these flows instead of directly regressing. This enables us to impose much stronger constraints encoding the conservation of the number of people. As a result, it significantly boosts performance without requiring a more complex architecture. Furthermore, it also enables us to exploit the correlation between people flow and optical flow to further improve the results. We will demonstrate that we consistently outperform state-of-the-art methods on five benchmark datasets.

Motivation & Objective

• To improve crowd counting in crowded scenes by modeling temporal dynamics more effectively than current density-based methods.
• To enforce strong temporal consistency through people flow conservation, ensuring no artificial creation or loss of individuals across frames.
• To leverage the correlation between optical flow and people flow to further regularize predictions and improve robustness.
• To demonstrate that flow-based estimation outperforms direct density regression even with standard network architectures.
• To validate the method on multiple benchmark datasets and show consistent gains across diverse crowded scenes.

Proposed method

• The method partitions the image into a grid and regresses ten types of flows per grid cell: nine toward neighboring cells and one toward the outside, with one flow representing people staying in place.
• People density at each grid location is computed by summing all incoming flows, enabling conservation of people across time via flow continuity.
• A novel loss function enforces strict conservation of people by ensuring the net flow into each cell matches the change in density, derived from Eq. (1) and (3).
• The approach uses a CAN-based network to regress flows from two consecutive frames, with additional supervision from optical flow to align motion patterns.
• The method supports both image-plane and ground-plane flow estimation, with the latter improving performance when camera calibration is available.
• Ablation studies compare flow-based regression against direct density regression, temporal averaging, and weak conservation constraints.

Experimental results

Research questions

• RQ1 Can modeling people flows between consecutive frames improve crowd counting accuracy compared to direct density regression?
• RQ2 Does enforcing strict conservation of people across frames lead to more robust and accurate density estimates?
• RQ3 Can the correlation between optical flow and people flow be exploited to further improve performance?
• RQ4 Does the proposed flow-based formulation outperform existing methods without requiring deeper or more complex network architectures?
• RQ5 How does the method perform across diverse datasets and under varying conditions such as camera calibration?

Key findings

• The proposed method achieves a 21.5% reduction in MAE on the UCSD dataset compared to the baseline, with MAE dropping from 0.98 to 0.81.
• On the FDST dataset, the method reduces RMSE from 1.26 (baseline) to 1.07, demonstrating consistent improvement across metrics.
• The ablation study confirms that flow-based estimation with strong conservation constraints (OURS-COMBI) outperforms weak constraints (WEAK) and direct density regression (BASELINE).
• The combination of flow regression and optical flow supervision (OURS-COMBI) yields the best performance, showing that motion correlation enhances accuracy.
• The method consistently outperforms state-of-the-art approaches on five benchmark datasets, including CrowdFlow, FDST, and UCSD.
• When applied in the ground plane using camera calibration, performance improves further, as shown in the supplementary material.