[Paper Review] Focus on Semantic Consistency for Cross-domain Crowd Understanding

Tao Han, Junyu Gao+2 more|arXiv (Cornell University)|Feb 20, 2020

Video Surveillance and Tracking Methods28 references4 citations

TL;DR

This paper proposes a domain adaptation framework for cross-domain crowd counting that enhances semantic consistency between synthetic and real-world crowd scenes. By introducing a semantic extractor to focus on crowd regions and using adversarial learning to align high-level features, the method reduces background estimation errors and achieves state-of-the-art performance on three real-world datasets, with MAE improvements of 31.3% over baseline models.

ABSTRACT

For pixel-level crowd understanding, it is time-consuming and laborious in data collection and annotation. Some domain adaptation algorithms try to liberate it by training models with synthetic data, and the results in some recent works have proved the feasibility. However, we found that a mass of estimation errors in the background areas impede the performance of the existing methods. In this paper, we propose a domain adaptation method to eliminate it. According to the semantic consistency, a similar distribution in deep layer's features of the synthetic and real-world crowd area, we first introduce a semantic extractor to effectively distinguish crowd and background in high-level semantic information. Besides, to further enhance the adapted model, we adopt adversarial learning to align features in the semantic space. Experiments on three representative real datasets show that the proposed domain adaptation scheme achieves the state-of-the-art for cross-domain counting problems.

Motivation & Objective

• To address the challenge of high background estimation errors in cross-domain crowd counting when using synthetic data.
• To reduce the domain gap between synthetic (GCC) and real-world (Shanghai Tech, UCF-QNRF) crowd datasets.
• To improve crowd counting performance by focusing on semantic consistency in high-level features of crowd regions.
• To develop a domain adaptation framework that leverages both semantic supervision and adversarial feature alignment.
• To achieve state-of-the-art performance on low-density and high-density crowd counting benchmarks.

Proposed method

• A semantic extractor network is trained using crowd masks from the GCC dataset to generate high-level semantic labels for crowd regions.
• The framework uses a VGG16-based feature extractor with batch normalization to extract deep features from both source (synthetic) and target (real) images.
• Adversarial learning is applied via a feature discriminator to align the deep feature distributions of source and target domains in the semantic space.
• A multi-task loss function combines crowd counting loss, semantic segmentation loss, and adversarial loss to jointly optimize the model.
• Scene regularization is applied to select representative images from the large GCC dataset to avoid negative domain shift during training.
• The model is trained using Adam with a low learning rate of 1e-5 and a batch size of 8 on 480×640 cropped inputs.

Experimental results

Research questions

• RQ1 Can focusing on semantic consistency in high-level features reduce domain gap in cross-domain crowd counting?
• RQ2 Does leveraging crowd-specific semantic supervision improve generalization from synthetic to real-world data?
• RQ3 Can adversarial feature alignment in the semantic space outperform image-style transfer methods like SE Cycle GAN?
• RQ4 How does the proposed method perform on both low-density and high-density crowd datasets?
• RQ5 To what extent does the method reduce background estimation errors compared to existing domain adaptation approaches?

Key findings

• The proposed method achieves a MAE of 16.9 and MSE of 24.7 on the Shanghai Tech Part B dataset, representing a 31.3% and 26.7% reduction in error compared to the baseline model.
• On the UCF-QNRF dataset, the method outperforms the state-of-the-art SE Cycle GAN in both MAE and MSE, demonstrating superior generalization to highly congested scenes.
• Visual results show that density maps generated with domain adaptation are significantly closer to ground truth than those without, especially in background regions.
• The method improves PSNR and SSIM scores, indicating better quality of predicted density maps.
• The ablation study confirms that both the semantic extractor and adversarial alignment contribute to performance gains, with the semantic component being critical for reducing background noise.
• The framework generalizes well across datasets, achieving SOTA results on both low-density (Shanghai Tech) and high-density (UCF-QNRF) crowd counting benchmarks.