[Paper Review] Motion-guided Non-local Spatial-Temporal Network for Video Crowd Counting

Haoyue Bai, S.-H. Gary Chan|arXiv (Cornell University)|Apr 28, 2021

Video Surveillance and Tracking Methods32 references4 citations

TL;DR

This paper proposes Monet, a motion-guided non-local spatial-temporal network for video crowd counting that leverages optical flow to guide coarse segmentation of person regions and employs a non-local network to capture both short- and long-range spatial-temporal context. Monet achieves state-of-the-art performance on VidCrowd, Mall, and UCSD datasets, with a 1.54 MAE on UCSD and 1.17 MAE on the challenging VidCrowd dataset, while introducing a new large-scale video crowd counting dataset, VidCrowd, with 9,000 high-resolution frames and 1.15 million head annotations.

ABSTRACT

We study video crowd counting, which is to estimate the number of objects (people in this paper) in all the frames of a video sequence. Previous work on crowd counting is mostly on still images. There has been little work on how to properly extract and take advantage of the spatial-temporal correlation between neighboring frames in both short and long ranges to achieve high estimation accuracy for a video sequence. In this work, we propose Monet, a novel and highly accurate motion-guided non-local spatial-temporal network for video crowd counting. Monet first takes people flow (motion information) as guidance to coarsely segment the regions of pixels where a person may be. Given these regions, Monet then uses a non-local spatial-temporal network to extract spatial-temporally both short and long-range contextual information. The whole network is finally trained end-to-end with a fused loss to generate a high-quality density map. Noting the scarcity and low quality (in terms of resolution and scene diversity) of the publicly available video crowd datasets, we have collected and built a large-scale video crowd counting datasets, VidCrowd, to contribute to the community. VidCrowd contains 9,000 frames of high resolution (2560 x 1440), with 1,150,239 head annotations captured in different scenes, crowd density and lighting in two cities. We have conducted extensive experiments on the challenging VideoCrowd and two public video crowd counting datasets: UCSD and Mall. Our approach achieves substantially better performance in terms of MAE and MSE as compared with other state-of-the-art approaches.

Motivation & Objective

• To address the lack of effective modeling of spatial-temporal correlations in video crowd counting, especially long-range dependencies across frames.
• To improve estimation accuracy by integrating motion information (optical flow) as a spatial prior for person region localization.
• To develop a deep learning framework that jointly captures local and non-local spatial-temporal context for density map regression.
• To alleviate the scarcity and low quality of existing video crowd counting datasets by creating and releasing a new large-scale, high-resolution dataset, VidCrowd.
• To establish a new benchmark for video crowd counting with diverse scenes, lighting, and crowd densities.

Proposed method

• The method begins with motion estimation using optical flow to generate motion vectors that segment the video frame into regions likely to contain people, serving as a spatial prior.
• A non-local spatial-temporal network is applied to consecutive frames, guided by the motion-based segmentation, to extract both local (short-range) and non-local (long-range) contextual features across space and time.
• The network uses a cascaded refinement module to progressively refine the predicted density map, integrating motion guidance and non-local context.
• A fused loss function combines multiple supervision signals to train the entire network end-to-end, improving density map quality and counting accuracy.
• The framework is trained and evaluated on the newly collected VidCrowd dataset, which contains 9,000 high-resolution (2560×1440) frames with 1,150,239 head annotations across 20 diverse scenes.

Experimental results

Research questions

• RQ1 How can motion information (optical flow) be effectively used to guide the localization of person regions in video frames for improved crowd counting?
• RQ2 To what extent can a non-local spatial-temporal network capture long-range dependencies in video sequences to enhance crowd counting accuracy?
• RQ3 Can a unified deep learning framework that combines motion guidance and non-local modeling outperform existing video-based crowd counting methods on challenging benchmarks?
• RQ4 How does the proposed Monet model perform on a new, large-scale, high-resolution video crowd counting dataset with diverse real-world conditions?

Key findings

• Monet achieves a 1.54 MAE and 2.02 MSE on the UCSD dataset, outperforming all prior video-based methods and matching the state-of-the-art performance on this benchmark.
• On the newly introduced VidCrowd dataset, Monet achieves a 1.17 MAE and 1.45 MSE, significantly outperforming all compared methods, including MCNN, CSRNet, SACANet, and ConvLSTM.
• Ablation studies confirm that both motion guidance and the non-local spatial-temporal module are essential: removing either leads to a significant drop in performance, validating their complementary roles.
• The proposed Monet framework achieves state-of-the-art results on the Mall dataset, with a 2.10 MAE and 7.60 MSE, surpassing all existing video-based counting models.
• The new VidCrowd dataset, containing 9,000 high-resolution frames and 1,150,239 head annotations, provides a more diverse and challenging benchmark than existing datasets, with varied scenes, lighting, and crowd densities.