[Paper Review] UP-DETR: Unsupervised Pre-training for Object Detection with Transformers

Zhigang Dai, Bolun Cai+2 more|arXiv (Cornell University)|Nov 18, 2020

Advanced Neural Network ApplicationsComputer Science45 references64 citations

TL;DR

UP-DETR pre-trains the DETR transformer using a novel random query patch detection task on unlabeled images, freezing the CNN backbone during pre-training to boost DETR’s convergence and performance on object detection, one-shot detection, and panoptic segmentation.

ABSTRACT

DEtection TRansformer (DETR) for object detection reaches competitive performance compared with Faster R-CNN via a transformer encoder-decoder architecture. However, trained with scratch transformers, DETR needs large-scale training data and an extreme long training schedule even on COCO dataset. Inspired by the great success of pre-training transformers in natural language processing, we propose a novel pretext task named random query patch detection in Unsupervised Pre-training DETR (UP-DETR). Specifically, we randomly crop patches from the given image and then feed them as queries to the decoder. The model is pre-trained to detect these query patches from the input image. During the pre-training, we address two critical issues: multi-task learning and multi-query localization. (1) To trade off classification and localization preferences in the pretext task, we find that freezing the CNN backbone is the prerequisite for the success of pre-training transformers. (2) To perform multi-query localization, we develop UP-DETR with multi-query patch detection with attention mask. Besides, UP-DETR also provides a unified perspective for fine-tuning object detection and one-shot detection tasks. In our experiments, UP-DETR significantly boosts the performance of DETR with faster convergence and higher average precision on object detection, one-shot detection and panoptic segmentation. Code and pre-training models: https://github.com/dddzg/up-detr.

Motivation & Objective

• Motivate improving DETR by pre-training transformers rather than starting from scratch on limited data.
• Introduce a self-supervised pretext task aligned with DETR’s localization focus: random query patch detection.
• Ensure stable pre-training by freezing the CNN backbone and balancing classification vs localization features.
• Enable a unified fine-tuning path for object detection and one-shot detection using the same pre-trained model.
• Explore extensions to multi-query patch detection and attention masking to mimic NMS-like behavior in DETR.

Proposed method

• Pre-train the transformer encoder-decoder with a random query patch detection task on unlabeled images.
• Use a CNN backbone to extract patch features; feed query patches and object queries into a transformer decoder to predict patch bounding boxes.
• Employ Hungarian matching loss combining classification, box regression (L1 + IoU), and an optional patch reconstruction loss to preserve localization/features.
• Freeze the CNN backbone during pre-training to maintain feature discrimination and enable effective localization learning.
• Extend to multi-query patch detection by grouping object queries and applying an attention mask to control interactions between groups.

Figure 1 : The PASCAL VOC learning curves ( $\textrm{AP}_{50}$ ) of DETR and UP-DETR with the ResNet-50 backbone. Here, they are trained on trainval07+12 and evaluated on test2007 . We plot the short and long training schedules, and the learning rate is reduced at 100 and 200 epochs respectively for

Experimental results

Research questions

• RQ1 Does unsupervised pre-training of the DETR transformer improve convergence speed and detection accuracy on VOC/COCO datasets compared to training from scratch?
• RQ2 Can a random query patch detection pretext task leverage DETR’s localization focus more effectively with a frozen CNN backbone?
• RQ3 Does multi-query patch detection and attention masking better reflect inter-query competition and improve downstream tasks like one-shot detection and panoptic segmentation?
• RQ4 How well does UP-DETR transfer to one-shot detection and panoptic segmentation compared to DETR?
• RQ5 What is the impact of patch feature reconstruction on preserving classification-style features during localization pre-training?

Key findings

• UP-DETR converges faster and achieves higher AP than DETR on VOC and COCO after both short and long training schedules.
• On PASCAL VOC, UP-DETR with a frozen backbone yields up to +6.2 AP (150 epochs) and +7.5 AP (300 epochs) over DETR, approaching Faster R-CNN performance.
• On COCO, UP-DETR with 150 epochs slightly surpasses DETR and matches Faster R-CNN under comparable schedules, and with 300 epochs it exceeds DETR and slightly edges out Faster R-CNN (R50-FPN) in AP.
• One-shot detection results show UP-DETR greatly improves over DETR, achieving significant gains on both seen and unseen classes across VOC settings.
• Panoptic segmentation results indicate UP-DETR improves over DETR in PQ, SQ, and RQ metrics, translating to higher AP seg and related panoptic scores.
• Ablation shows multi-query patches (M=10) outperform single-query patches, and freezing the backbone plus patch feature reconstruction are beneficial for pre-training effectiveness.

(a) single-query patch (N=3, M=1)