[Paper Review] DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR

Shilong Liu, Feng Li+6 more|arXiv (Cornell University)|Jan 28, 2022

Anomaly Detection Techniques and Applications391 citations

TL;DR

DAB-DETR treats anchor boxes (x, y, w, h) as dynamically updated queries in DETR, modulating cross-attention with object scale and achieving state-of-the-art DETR-like performance on COCO at the same settings.

ABSTRACT

We present in this paper a novel query formulation using dynamic anchor boxes for DETR (DEtection TRansformer) and offer a deeper understanding of the role of queries in DETR. This new formulation directly uses box coordinates as queries in Transformer decoders and dynamically updates them layer-by-layer. Using box coordinates not only helps using explicit positional priors to improve the query-to-feature similarity and eliminate the slow training convergence issue in DETR, but also allows us to modulate the positional attention map using the box width and height information. Such a design makes it clear that queries in DETR can be implemented as performing soft ROI pooling layer-by-layer in a cascade manner. As a result, it leads to the best performance on MS-COCO benchmark among the DETR-like detection models under the same setting, e.g., AP 45.7\% using ResNet50-DC5 as backbone trained in 50 epochs. We also conducted extensive experiments to confirm our analysis and verify the effectiveness of our methods. Code is available at \url{https://github.com/SlongLiu/DAB-DETR}.

Motivation & Objective

• Motivate a better understanding of DETR queries and improve DETR training convergence.
• Introduce a query formulation where anchors are directly learned and used as queries in the Transformer decoder.
• Leverage object scale information to modulate cross-attention for more accurate feature pooling.
• Show that dynamic, layer-wise updating of anchor boxes yields performance gains over traditional DETR-style queries.
• Demonstrate state-of-the-art results among DETR-like models under the same settings on COCO.

Proposed method

• Propose 4D anchor boxes (x, y, w, h) as queries and generate positional queries via an MLP over sinusoidal embeddings.
• Use concatenated content and positional information in both self- and cross-attention, with a scale-conditioned cross-attention mechanism.
• Update anchors layer-by-layer by predicting relative deltas (Δx, Δy, Δw, Δh) with shared heads across layers.
• Modulate the Gaussian-like positional prior in attention with width and height information (w, h) to adapt to object scales.
• Introduce a temperature parameter T to tune the softness of the positional attention, using a vision-specific setting (T = 20).
• Provide an optional stronger variant (DAB-DETR-Deformable) and detailed ablations confirming the contribution of each component.

Experimental results

Research questions

• RQ1 How do anchor-based queries compare to traditional learnable queries in DETR in terms of convergence and accuracy?
• RQ2 Can dynamic, scale-aware anchor boxes as queries improve cross-attention and object localization across objects of varying sizes?
• RQ3 What is the impact of updating anchor boxes layer-by-layer on DETR-like detectors?
• RQ4 Does incorporating width/height-modulated attention and temperature tuning yield measurable gains on COCO?EDR?

Key findings

• DAB-DETR achieves 45.7 AP on COCO with ResNet-50 (DC5) trained for 50 epochs, outperforming several DETR-like models under the same settings.
• Using 4D anchor boxes as queries improves AP from 44.0 to 45.0 when comparing anchor box vs. anchor point formulations (ablation).
• Dynamic anchor updates across layers provide a 1.7 AP gain.
• Width/height-modulated attention and temperature tuning contribute to further gains, with ablations showing performance drops if these components are removed.
• With pattern embeddings, DAB-DETR variants surpass prior DETR-like methods across backbones, validating the effectiveness of the proposed design.
• Compared to baseline DETR and several variants, DAB-DETR demonstrates strong performance even against multi-scale DETR-like approaches.