[Paper Review] Distribution Matching for Crowd Counting

Boyu Wang, Huidong Liu+2 more|arXiv (Cornell University)|Sep 28, 2020

Video Surveillance and Tracking MethodsComputer Science60 references169 citations

TL;DR

DM-Count uses Optimal Transport to match the normalized predicted density map with the normalized ground-truth density map, avoiding Gaussian smoothing and achieving state-of-the-art results on multiple crowd counting datasets.

ABSTRACT

In crowd counting, each training image contains multiple people, where each person is annotated by a dot. Existing crowd counting methods need to use a Gaussian to smooth each annotated dot or to estimate the likelihood of every pixel given the annotated point. In this paper, we show that imposing Gaussians to annotations hurts generalization performance. Instead, we propose to use Distribution Matching for crowd COUNTing (DM-Count). In DM-Count, we use Optimal Transport (OT) to measure the similarity between the normalized predicted density map and the normalized ground truth density map. To stabilize OT computation, we include a Total Variation loss in our model. We show that the generalization error bound of DM-Count is tighter than that of the Gaussian smoothed methods. In terms of Mean Absolute Error, DM-Count outperforms the previous state-of-the-art methods by a large margin on two large-scale counting datasets, UCF-QNRF and NWPU, and achieves the state-of-the-art results on the ShanghaiTech and UCF-CC50 datasets. DM-Count reduced the error of the state-of-the-art published result by approximately 16%. Code is available at https://github.com/cvlab-stonybrook/DM-Count.

Motivation & Objective

• Motivate the study by showing Gaussian smoothing of annotations hurts generalization in crowd counting.
• Propose a distribution matching framework (DM-Count) that uses Optimal Transport to compare predicted and ground-truth density maps without Gaussian smoothing.
• Stabilize OT computation with a Total Variation loss and provide generalization bounds for the proposed losses.
• Demonstrate empirical improvements over prior methods on four large crowd counting benchmarks.

Proposed method

• Define the counting problem as distribution matching between the ground-truth dot annotations and the predicted density map.
• Use Optimal Transport to compute a transport-based loss between normalized ground-truth and predicted density maps.
• Introduce a counting loss based on the absolute difference in total mass to align counts.
• Incorporate a Total Variation loss to stabilize Sinkhorn-based OT optimization and improve low-density areas.
• Combine counting loss, OT loss, and TV loss into a single training objective with tunable weights.

Experimental results

Research questions

• RQ1 Can distribution matching via Optimal Transport improve crowd counting without Gaussian ground-truth smoothing?
• RQ2 How does OT-based loss compare to Gaussian-smoothed and Bayesian losses in terms of generalization and localization quality?
• RQ3 Do stability enhancements (Total Variation) improve training when using Sinkhorn approximations for OT?
• RQ4 What are the empirical gains of DM-Count on large-scale datasets compared to prior state-of-the-art methods?

Key findings

• DM-Count outperforms prior state-of-the-art methods on four datasets (UCF-QNRF, NWPU, ShanghaiTech, UCF-CC50) in MAE, RMSE, and NAE.
• On NWPU, DM-Count reduces published MAE and NAE by substantial margins (e.g., MAE from 105.4 to 88.4; NAE from 0.203 to 0.169).
• DM-Count additionally improves density map quality, achieving higher PSNR and SSIM than pixel-wise and Bayesian losses on key benchmarks.
• Theoretical analysis shows Gaussian smoothing ground truth annotations yields looser generalization bounds than the DM-Count approach.
• Ablation studies indicate the OT loss is the most impactful component, with TV providing stability benefits during training.
• DM-Count demonstrates robustness to annotation noise compared with Gaussian-smoothed or Bayesian losses.