[Paper Review] Crowd Counting and Density Estimation by Trellis Encoder-Decoder Network

Xiaolong Jiang, Zehao Xiao+5 more|arXiv (Cornell University)|Mar 3, 2019

Video Surveillance and Tracking MethodsComputer Science49 references78 citations

TL;DR

TEDnet proposes a trellis-style multi-path encoder-decoder with dense skip connections and a combinatorial loss to produce high-quality density maps and accurate crowd counts, achieving state-of-the-art results on multiple benchmarks.

ABSTRACT

Crowd counting has recently attracted increasing interest in computer vision but remains a challenging problem. In this paper, we propose a trellis encoder-decoder network (TEDnet) for crowd counting, which focuses on generating high-quality density estimation maps. The major contributions are four-fold. First, we develop a new trellis architecture that incorporates multiple decoding paths to hierarchically aggregate features at different encoding stages, which can handle large variations of objects. Second, we design dense skip connections interleaved across paths to facilitate sufficient multi-scale feature fusions and to absorb the supervision information. Third, we propose a new combinatorial loss to enforce local coherence and spatial correlation in density maps. By distributedly imposing this combinatorial loss on intermediate outputs, gradient vanishing can be largely alleviated for better back-propagation and faster convergence. Finally, our TEDnet achieves new state-of-the art performance on four benchmarks, with an improvement up to 14% in terms of MAE.

Motivation & Objective

• Motivate robust crowd counting under occlusion and scale variation by preserving spatial precision.
• Develop a network that maintains high-resolution localization while extracting semantic features.
• Improve density-map quality beyond patch-based methods through multi-path fusion and distributed supervision.
• Address gradient vanishing and map coherency via a novel combinatorial loss.
• Demonstrate state-of-the-art counting accuracy and density-map quality on standard benchmarks.

Proposed method

• Introduce a multi-scale encoder to extract scale-adaptive features with limited down-sampling.
• Design a multi-path decoder that hierarchically aggregates features along several decoding paths with dense skip connections.
• Apply distributed supervision by adding intermediate density map outputs with corresponding losses.
• Propose a combinatorial loss composed of Spatial Abstraction Loss (SAL) and Spatial Correlation Loss (SCL) to enforce local coherence and spatial correlation in density maps.
• Compute losses on full-resolution density maps rather than patches to preserve localization precision.
• Train end-to-end with Adam on full images, using fixed Gaussian ground-truth maps and online data augmentation.

Experimental results

Research questions

• RQ1 Can a trellis-like multi-path encoder-decoder improve pixel-wise density estimation for crowded scenes?
• RQ2 Does dense inter-path fusion enhance multi-scale feature aggregation and density-map quality?
• RQ3 Will distributed supervision and a combinatorial loss improve gradient flow and address gradient vanishing?
• RQ4 How does TEDnet perform on standard benchmarks in terms of MAE/MSE and density-map quality (PSNR/SSIM) compared to prior methods?

Key findings

• TEDnet achieves state-of-the-art performance on four benchmarks, with notable MAE improvements over prior methods.
• The multi-path decoder with dense skip connections yields better density-map quality (PSNR/SSIM) and counting accuracy.
• Distributed supervision reduces gradient vanishing and accelerates convergence.
• The combinatorial SAL/SCL loss improves density-map coherence and spatial correlation, further boosting counting performance.
• TEDnet generates full-resolution density maps from full images, avoiding boundary artifacts typical of patch-based methods.
• TEDnet maintains a lightweight parameter footprint while delivering superior density-map quality and counting accuracy.