[Paper Review] DETR for Crowd Pedestrian Detection

Matthieu Lin, Chuming Li+6 more|arXiv (Cornell University)|Dec 12, 2020

Advanced Neural Network Applications39 references38 citations

TL;DR

PED-DETR is a pedestrian-end-to-end detector that analyzes why DETR underperforms in crowds and introduces DQRF, V-Match, and Fast-KM to achieve state-of-the-art results on CrowdHuman and CityPersons.

ABSTRACT

Pedestrian detection in crowd scenes poses a challenging problem due to the heuristic defined mapping from anchors to pedestrians and the conflict between NMS and highly overlapped pedestrians. The recently proposed end-to-end detectors(ED), DETR and deformable DETR, replace hand designed components such as NMS and anchors using the transformer architecture, which gets rid of duplicate predictions by computing all pairwise interactions between queries. Inspired by these works, we explore their performance on crowd pedestrian detection. Surprisingly, compared to Faster-RCNN with FPN, the results are opposite to those obtained on COCO. Furthermore, the bipartite match of ED harms the training efficiency due to the large ground truth number in crowd scenes. In this work, we identify the underlying motives driving ED's poor performance and propose a new decoder to address them. Moreover, we design a mechanism to leverage the less occluded visible parts of pedestrian specifically for ED, and achieve further improvements. A faster bipartite match algorithm is also introduced to make ED training on crowd dataset more practical. The proposed detector PED(Pedestrian End-to-end Detector) outperforms both previous EDs and the baseline Faster-RCNN on CityPersons and CrowdHuman. It also achieves comparable performance with state-of-the-art pedestrian detection methods. Code will be released soon.

Motivation & Objective

• Assess why DETR and deformable DETR underperform on crowd pedestrian detection compared to Faster-RCNN with FPN.
• Propose a Dense Query and Rectified Attention Field decoder (DQRF) to improve DETR’s performance for pedestrians.
• Leverage visible-region annotations with V-Match to boost DETR-based pedestrian detection.
• Introduce a faster bipartite matching (Fast-KM) to make training DETR feasible on crowded datasets.
• Demonstrate state-of-the-art or competitive performance on CrowdHuman and CityPersons benchmarks.

Proposed method

• Analyze DETR’s decoder behavior in crowded pedestrian scenes to identify failure modes.
• Develop DQRF decoder to enable dense queries and a rectified, broader attention field.
• Introduce RF (Rectified Attention Field) to stabilize cross-attention across decoder layers.
• Propose V-Match to supervise full and visible pedestrian regions across layers without extra cost.
• Implement Fast-KM to accelerate the Hungarian matching step during training.

Experimental results

Research questions

• RQ1 Why do original DETR and deformable DETR underperform on crowd pedestrian detection compared to Faster-RCNN?
• RQ2 Can aDense Query and Rectified Attention Field decoder improve DETR’s detection of crowded pedestrians?
• RQ3 Does leveraging visible-region annotations via V-Match improve end-to-end pedestrian detection without extra cost?
• RQ4 How much speedup in bipartite matching training can be achieved with Fast-KM without sacrificing accuracy?

Key findings

• PED achieves improvements over deformable DETR and Faster-RCNN baselines on CrowdHuman and CityPersons.
• The proposed DQRF decoder significantly closes the gap between DETR-based detectors and Faster-RCNN in crowded scenes.
• V-Match provides visible-region supervision gains with zero extra cost.
• Fast-KM delivers up to 10x speed-up for bipartite matching during training.
• PED attains competitive results with state-of-the-art methods on challenging benchmarks.
• Dense queries and rectified attention fields reduce missed detections in crowded occluded settings.