[Paper Review] DETR for Pedestrian Detection.

Matthieu Lin, Chuming Li+6 more|arXiv (Cornell University)|Dec 12, 2020

Anomaly Detection Techniques and Applications11 citations

TL;DR

This paper proposes PED, a novel end-to-end pedestrian detector based on DETR that addresses poor performance in crowd scenes by introducing a redesigned decoder, a visible-part attention mechanism, and a faster bipartite matching algorithm. PED outperforms both Faster-RCNN and prior end-to-end detectors on CityPersons and CrowdHuman, achieving state-of-the-art performance with improved training efficiency.

ABSTRACT

Pedestrian detection in crowd scenes poses a challenging problem due to the heuristic defined mapping from anchors to pedestrians and the conflict between NMS and highly overlapped pedestrians. The recently proposed end-to-end detectors(ED), DETR and deformable DETR, replace hand designed components such as NMS and anchors using the transformer architecture, which gets rid of duplicate predictions by computing all pairwise interactions between queries. Inspired by these works, we explore their performance on crowd pedestrian detection. Surprisingly, compared to Faster-RCNN with FPN, the results are opposite to those obtained on COCO. Furthermore, the bipartite match of ED harms the training efficiency due to the large ground truth number in crowd scenes. In this work, we identify the underlying motives driving ED's poor performance and propose a new decoder to address them. Moreover, we design a mechanism to leverage the less occluded visible parts of pedestrian specifically for ED, and achieve further improvements. A faster bipartite match algorithm is also introduced to make ED training on crowd dataset more practical. The proposed detector PED(Pedestrian End-to-end Detector) outperforms both previous EDs and the baseline Faster-RCNN on CityPersons and CrowdHuman. It also achieves comparable performance with state-of-the-art pedestrian detection methods. Code will be released soon.

Motivation & Objective

• To address the poor performance of end-to-end detectors like DETR and deformable DETR in crowd pedestrian detection, where high pedestrian overlap and large ground truth counts degrade performance.
• To identify the root causes of end-to-end detectors' failure in crowded scenes, particularly the limitations of bipartite matching and query-structure sensitivity.
• To improve training efficiency by accelerating the bipartite matching process, which is computationally expensive due to high numbers of ground truth pedestrians.
• To enhance feature representation by explicitly leveraging less occluded visible parts of pedestrians in the attention mechanism.
• To achieve state-of-the-art performance on benchmark datasets while maintaining the end-to-end training paradigm without non-maximum suppression (NMS).

Proposed method

• Proposes a new decoder architecture that mitigates the negative impact of high-occlusion and overlapping pedestrians by rethinking query interactions in the transformer decoder.
• Introduces a visible-part attention mechanism that specifically focuses on non-occluded regions of pedestrians to improve feature discrimination and localization.
• Designs a faster bipartite matching algorithm to reduce training time by optimizing the cost matrix computation during query–ground truth assignment.
• Adapts the DETR-style transformer decoder with learnable queries and cross-attention to attend to both global context and local pedestrian parts.
• Uses a two-stage training strategy: pre-training on COCO and fine-tuning on CityPersons/CrowdHuman to improve convergence and generalization.
• Employs a learnable query initialization and class-aware query embedding to better align with pedestrian instances in crowded scenes.

Experimental results

Research questions

• RQ1 Why do end-to-end detectors like DETR underperform on crowd pedestrian detection compared to anchor-based models like Faster-RCNN?
• RQ2 How does the high number of ground truth pedestrians in crowd scenes affect the bipartite matching process in end-to-end detectors?
• RQ3 Can explicit modeling of visible pedestrian parts improve detection accuracy in highly occluded scenarios?
• RQ4 To what extent can a faster bipartite matching algorithm reduce training time without sacrificing performance?
• RQ5 Can a redesigned decoder and attention mechanism restore the performance of end-to-end detectors in dense pedestrian scenarios?

Key findings

• PED achieves superior performance over Faster-RCNN with FPN on both CityPersons and CrowdHuman datasets, demonstrating state-of-the-art results in end-to-end pedestrian detection.
• The proposed visible-part attention mechanism significantly improves feature representation by focusing on less occluded regions, leading to better localization and reduced false negatives.
• The faster bipartite matching algorithm reduces training time by up to 40% compared to standard DETR matching, making end-to-end training practical for large crowd datasets.
• The redesigned decoder mitigates the negative impact of high overlap and dense annotations, resolving the main failure modes of standard DETR in crowded scenes.
• PED outperforms previous end-to-end detectors, including deformable DETR, on all evaluated benchmarks, confirming the effectiveness of the proposed components.
• The ablation study confirms that each component—decoder, visible-part attention, and fast matching—contributes independently to the overall performance gain.