[Paper Review] NMS Strikes Back

Jeffrey Ouyang-Zhang, Jang Hyun Cho+2 more|arXiv (Cornell University)|Dec 12, 2022

Advanced Neural Network Applications6 citations

TL;DR

This paper proposes DETA, a detection transformer that replaces end-to-end one-to-one Hungarian matching with traditional one-to-many IoU-based label assignment and non-maximum suppression (NMS), achieving 50.2 mAP on COCO in just 12 epochs with ResNet50. The method outperforms both standard end-to-end transformers and state-of-the-art NMS-based detectors by leveraging more positive samples and improved training stability, demonstrating that bipartite matching is not essential for high performance in detection transformers.

ABSTRACT

Detection Transformer (DETR) directly transforms queries to unique objects by using one-to-one bipartite matching during training and enables end-to-end object detection. Recently, these models have surpassed traditional detectors on COCO with undeniable elegance. However, they differ from traditional detectors in multiple designs, including model architecture and training schedules, and thus the effectiveness of one-to-one matching is not fully understood. In this work, we conduct a strict comparison between the one-to-one Hungarian matching in DETRs and the one-to-many label assignments in traditional detectors with non-maximum supervision (NMS). Surprisingly, we observe one-to-many assignments with NMS consistently outperform standard one-to-one matching under the same setting, with a significant gain of up to 2.5 mAP. Our detector that trains Deformable-DETR with traditional IoU-based label assignment achieved 50.2 COCO mAP within 12 epochs (1x schedule) with ResNet50 backbone, outperforming all existing traditional or transformer-based detectors in this setting. On multiple datasets, schedules, and architectures, we consistently show bipartite matching is unnecessary for performant detection transformers. Furthermore, we attribute the success of detection transformers to their expressive transformer architecture. Code is available at https://github.com/jozhang97/DETA.

Motivation & Objective

• To investigate whether one-to-one bipartite matching is truly necessary for high-performing detection transformers.
• To compare the effectiveness of one-to-many label assignment with traditional NMS-based training versus one-to-one Hungarian matching in end-to-end detectors.
• To determine whether the success of detection transformers stems from their architecture or from the training objective.
• To develop a simpler, more efficient training objective that maintains or improves performance without requiring complex matching algorithms.
• To demonstrate that traditional training objectives can outperform end-to-end matching in modern transformer-based detectors.

Proposed method

• Replace the one-to-one Hungarian matching loss in Deformable-DETR with a one-to-many IoU-based label assignment in both first-stage and second-stage heads.
• Use standard NMS for post-processing to remove overlapping predictions, reintroducing a key component from traditional detectors.
• Apply an object balancing technique to stabilize training, especially for small objects, by limiting the number of positive predictions per ground truth object.
• Maintain the same transformer encoder and decoder architecture as Deformable-DETR, differing only in the training objective to ensure fair comparison.
• Use a fixed IoU threshold (e.g., 0.5) to assign positive labels to proposals or queries that exceed the threshold with any ground truth box.
• Train the model under the standard 1× schedule (12 epochs) with ResNet50 backbone to evaluate efficiency and performance.

Experimental results

Research questions

• RQ1 Does one-to-many label assignment with NMS outperform one-to-one Hungarian matching in detection transformers under identical architectural and training conditions?
• RQ2 What are the key factors enabling one-to-many assignment to achieve superior performance compared to bipartite matching?
• RQ3 Is the success of detection transformers primarily due to their transformer architecture or their training objective?
• RQ4 Can a simpler, traditional training objective replace complex end-to-end matching without sacrificing performance?
• RQ5 How does object balancing affect model convergence and performance, especially for small objects?

Key findings

• DETA achieves 50.2 mAP on COCO with ResNet50 and a 1× schedule (12 epochs), outperforming the best NMS-based detector (CenterNet2 at 42.9 mAP) and a strong end-to-end detector (DINO at 49.4 mAP).
• Switching from one-to-one Hungarian matching to one-to-many IoU-based assignment in Deformable-DETR yields a significant 2.5 mAP improvement.
• Object balancing improves mAP by 1.4 points, with the largest gains for small objects (+2.7 AP for small objects), indicating it mitigates sampling bias in standard matching.
• The method converges faster and achieves better performance than end-to-end detectors, even with a shorter training schedule.
• Experiments across multiple datasets (COCO, LVIS), architectures (DETR, Deformable-DETR), and backbones confirm consistent performance gains.
• Ablation studies show that self-attention in the decoder is not necessary for DETA, while cross-attention remains critical, suggesting that the model benefits from localized, query-based reasoning rather than global optimization.