[Paper Review] Top-Down Feedback for Crowd Counting Convolutional Neural Network

Deepak Babu Sam, R. Venkatesh Babu|arXiv (Cornell University)|Jul 24, 2018

Video Surveillance and Tracking MethodsComputer Science72 citations

TL;DR

The paper introduces a top-down feedback mechanism that gates bottom-up CNN activations to correct crowd density predictions, improving accuracy across major datasets.

ABSTRACT

Counting people in dense crowds is a demanding task even for humans. This is primarily due to the large variability in appearance of people. Often people are only seen as a bunch of blobs. Occlusions, pose variations and background clutter further compound the difficulty. In this scenario, identifying a person requires larger spatial context and semantics of the scene. But the current state-of-the-art CNN regressors for crowd counting are feedforward and use only limited spatial context to detect people. They look for local crowd patterns to regress the crowd density map, resulting in false predictions. Hence, we propose top-down feedback to correct the initial prediction of the CNN. Our architecture consists of a bottom-up CNN along with a separate top-down CNN to generate feedback. The bottom-up network, which regresses the crowd density map, has two columns of CNN with different receptive fields. Features from various layers of the bottom-up CNN are fed to the top-down network. The feedback, thus generated, is applied on the lower layers of the bottom-up network in the form of multiplicative gating. This masking weighs activations of the bottom-up network at spatial as well as feature levels to correct the density prediction. We evaluate the performance of our model on all major crowd datasets and show the effectiveness of top-down feedback.

Motivation & Objective

• Motivate the need for high-level scene context to correct density predictions in crowd counting.
• Propose a two-path architecture with a bottom-up density regressor and a top-down feedback generator.
• Demonstrate that multiplicative gating from top-down feedback improves counting accuracy across datasets.
• Show ablations to validate the effectiveness and reliability of the feedback mechanism.

Proposed method

• Bottom-up CNN regressor with two columns of different receptive fields to predict a density map.
• Top-down CNN generates feedback from high-level features of the bottom-up network.
• Feedback is applied as multiplicative gating to lower-layer activations of the bottom-up CNN.
• Training is staged: first train bottom-up CNN, then train top-down network with count loss and L1 regularization on gate features.
• Final density map is produced after applying gating, with standard L2 loss for bottom-up training and count loss for top-down training.
• Evaluation uses MAE and MSE across four datasets; density maps are downsampled to 1/4 resolution due to pooling.

Experimental results

Research questions

• RQ1 Can high-level scene context via a top-down module reduce false detections in dense crowds?
• RQ2 Does multiplicative gating of bottom-up activations improve crowd density estimation compared to a baseline bottom-up CNN?
• RQ3 Is the top-down feedback framework robust across datasets with varying density and viewpoint?
• RQ4 How does the top-down approach compare in parameter efficiency to other multi-column networks?

Key findings

• TDF-CNN achieves lower MAE and MSE than many baselines on Shanghaitech Part A and Part B, with fewer parameters.
• Ablation shows bottom-up CNN without feedback has MAE of 147.4 on Shanghaitech Part A, which drops to 97.5 with top-down feedback.
• Top-down feedback remains effective even with a single 9×9 column, reducing MAE by 21.4%.
• Gate feature maps act as sparsifying masks, selectively dampening spurious activations and preserving legitimate responses.
• On UCF CC 50, TDF-CNN achieves 354.7 MAE and 491.4 MSE with 0.13M parameters, competitive with methods having more parameters.
• On WorldExpo’10, the model attains superior average MAE and scene-wise MAE in several sequences, illustrating dataset-wide benefits.