[Paper Review] Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity

Byungseok Roh, JaeWoong Shin+2 more|arXiv (Cornell University)|Nov 29, 2021

Advanced Neural Network ApplicationsComputer Science28 references89 citations

TL;DR

Sparse DETR sparsifies encoder tokens with learnable criteria to reduce computation, achieving better or comparable AP with substantial speedups, including 10% token usage on COCO with notable gains over Deformable DETR.

ABSTRACT

DETR is the first end-to-end object detector using a transformer encoder-decoder architecture and demonstrates competitive performance but low computational efficiency on high resolution feature maps. The subsequent work, Deformable DETR, enhances the efficiency of DETR by replacing dense attention with deformable attention, which achieves 10x faster convergence and improved performance. Deformable DETR uses the multiscale feature to ameliorate performance, however, the number of encoder tokens increases by 20x compared to DETR, and the computation cost of the encoder attention remains a bottleneck. In our preliminary experiment, we observe that the detection performance hardly deteriorates even if only a part of the encoder token is updated. Inspired by this observation, we propose Sparse DETR that selectively updates only the tokens expected to be referenced by the decoder, thus help the model effectively detect objects. In addition, we show that applying an auxiliary detection loss on the selected tokens in the encoder improves the performance while minimizing computational overhead. We validate that Sparse DETR achieves better performance than Deformable DETR even with only 10% encoder tokens on the COCO dataset. Albeit only the encoder tokens are sparsified, the total computation cost decreases by 38% and the frames per second (FPS) increases by 42% compared to Deformable DETR. Code is available at https://github.com/kakaobrain/sparse-detr

Motivation & Objective

• Motivate reducing computational bottlenecks in end-to-end DETR-based detectors by sparsifying encoder tokens.
• Propose learnable criteria to select salient encoder tokens.
• Show that auxiliary encoder losses stabilize training and improve accuracy.
• Demonstrate improved efficiency and performance over Deformable DETR on COCO with multi-scale features.

Proposed method

• Introduce a saliency-based, learnable token sparsification scheme for the encoder.
• Use a scoring network to predict Decoder cross-Attention Map (DAM) as saliency signals.
• Define a top-rho sparsified token set for encoder updates per layer.
• Apply an encoder auxiliary loss to selected tokens to stabilize training and boost performance.
• Adopt top-k decoder queries derived from encoder outputs to refine predictions.
• Evaluate with Swin-T and ResNet-50 backbones on COCO 2017 val, comparing against DETR, Deformable DETR, PnP-DETR, and Faster R-CNN-FPN.

Experimental results

Research questions

• RQ1 Can encoder token sparsification in DETR-based detectors reduce computation without sacrificing detection accuracy?
• RQ2 Do saliency criteria based on objectness and decoder cross-attention (DAM) better identify tokens to update than random or objectness alone?
• RQ3 Does an encoder auxiliary loss improve convergence and allow deeper encoder stacks under sparsity?
• RQ4 How does Sparse DETR perform with multi-scale features (e.g., Swin-T) compared to Deformable DETR under varied sparsity levels?

Key findings

• Sparse DETR achieves competitive AP with significantly reduced computation, including 38% lower FLOPs and 42% higher FPS versus Deformable DETR.
• Using only 10% encoder tokens with DAM-based sparsification, Sparse DETR outperforms many baselines and rivals Deformable DETR+ on Swin-T backbones.
• DAM-based token selection consistently outperforms Objectness Score (OS) and random sampling across backbones and sparsity levels.
• Encoder auxiliary loss enables deeper encoders (e.g., 12 layers) with stable training and improved detection performance.
• Dynamic sparsification during inference maintains robust performance across keeping-ratio settings, outperforming similar dynamic strategies in some baselines.
• With Swin-T backbone, Sparse DETR at 10% encoder tokens yields substantial efficiency gains (12-82% token-level reduction) while preserving or improving AP, especially at larger object scales.