[Paper Review] CrowdHuman: A Benchmark for Detecting Human in a Crowd

Shuai Shao, Zijian Zhao+5 more|arXiv (Cornell University)|Apr 30, 2018

Video Surveillance and Tracking Methods23 references495 citations

TL;DR

The paper introduces CrowdHuman, a large-scale, richly annotated dataset for pedestrian detection in crowded scenes, with 470k instances, 22.6 persons per image, and three bounding-box types per person, plus shows strong cross-dataset generalization when used for pretraining.

ABSTRACT

Human detection has witnessed impressive progress in recent years. However, the occlusion issue of detecting human in highly crowded environments is far from solved. To make matters worse, crowd scenarios are still under-represented in current human detection benchmarks. In this paper, we introduce a new dataset, called CrowdHuman, to better evaluate detectors in crowd scenarios. The CrowdHuman dataset is large, rich-annotated and contains high diversity. There are a total of $470K$ human instances from the train and validation subsets, and $~22.6$ persons per image, with various kinds of occlusions in the dataset. Each human instance is annotated with a head bounding-box, human visible-region bounding-box and human full-body bounding-box. Baseline performance of state-of-the-art detection frameworks on CrowdHuman is presented. The cross-dataset generalization results of CrowdHuman dataset demonstrate state-of-the-art performance on previous dataset including Caltech-USA, CityPersons, and Brainwash without bells and whistles. We hope our dataset will serve as a solid baseline and help promote future research in human detection tasks.

Motivation & Objective

• Address the challenge of detecting humans in crowded scenes with heavy occlusion.
• Provide a large, diverse dataset to better represent crowd occlusions than existing benchmarks.
• Offer three bounding-box annotations per person (head, visible region, full body) to support occlusion-aware detection.
• Demonstrate cross-dataset generalization and the utility of CrowdHuman as pretraining data for other benchmarks.

Proposed method

• Collect and annotate a diverse set of crowd scenes from web images (~60k candidates, ~25k final) across 15k training, 4,370 validation, and 5,000 testing images.
• Annotate each human with full-body, visible-region, and head bounding boxes, with double-checks for quality.
• Provide a rich set of statistics on density, occlusion, and pairwise/triadic overlaps to characterize crowd difficulty.
• Evaluate baseline detectors (FPN with Faster R-CNN, and RetinaNet) using mMR and AP metrics, with anchor ratios adapted for full/visible/head tasks.
• Conduct cross-dataset experiments: pretrain on CrowdHuman then fine-tune on Caltech, CityPersons, COCOPersons, and Brainwash to assess generalization.

Experimental results

Research questions

• RQ1 How does CrowdHuman performance compare to existing datasets in crowded scenarios?
• RQ2 Can CrowdHuman serve as effective pretraining data to improve detection on Caltech, CityPersons, COCOPersons, and Brainwash?
• RQ3 What benefits do the three bounding-box annotations provide for detecting full bodies, visible regions, and heads in crowds?
• RQ4 How well do detectors pretrained on CrowdHuman generalize to various pedestrian and head-detection benchmarks?

Key findings

• CrowdHuman contains about 470k person instances in the train+validation subsets, across 15,000 training images with an average of 22.6 persons per image.
• The dataset provides three bounding-box types for each person: head, visible-body, and full-body.
• Baseline detectors (FPN and RetinaNet) show strong performance differences, with FPN generally outperforming RetinaNet on these tasks.
• Cross-dataset pretraining on CrowdHuman improves performance on Caltech (mMR 8.81 vs 10.08 for Caltech baseline), CityPersons (mMR 21.18 vs 14.81 top-quoted), and Brainwash (mMR 17.24 vs 19.77).
• Pretraining on CrowdHuman before finetuning on COCOPersons yields AP of 85.02 and mMR 39.79, compared with 83.83 AP and 41.89 mMR when trained on COCOPersons alone.
• The CrowdHuman pretraining also improves CityPersons results when followed by fine-tuning (e.g., mMR 10.67 when finetuned from CrowdHuman to CityPersons).