[Paper Review] Toward Transformer-Based Object Detection

Josh Beal, Eric Kim+4 more|arXiv (Cornell University)|Dec 17, 2020

Advanced Neural Network ApplicationsComputer Science40 references140 citations

TL;DR

ViT-FRCNN shows that a Vision Transformer backbone can be repurposed with a Faster R-CNN style detector to achieve competitive COCO detection results and better out-of-domain generalization, highlighting the benefits of large-scale pretraining for detection tasks.

ABSTRACT

Transformers have become the dominant model in natural language processing, owing to their ability to pretrain on massive amounts of data, then transfer to smaller, more specific tasks via fine-tuning. The Vision Transformer was the first major attempt to apply a pure transformer model directly to images as input, demonstrating that as compared to convolutional networks, transformer-based architectures can achieve competitive results on benchmark classification tasks. However, the computational complexity of the attention operator means that we are limited to low-resolution inputs. For more complex tasks such as detection or segmentation, maintaining a high input resolution is crucial to ensure that models can properly identify and reflect fine details in their output. This naturally raises the question of whether or not transformer-based architectures such as the Vision Transformer are capable of performing tasks other than classification. In this paper, we determine that Vision Transformers can be used as a backbone by a common detection task head to produce competitive COCO results. The model that we propose, ViT-FRCNN, demonstrates several known properties associated with transformers, including large pretraining capacity and fast fine-tuning performance. We also investigate improvements over a standard detection backbone, including superior performance on out-of-domain images, better performance on large objects, and a lessened reliance on non-maximum suppression. We view ViT-FRCNN as an important stepping stone toward a pure-transformer solution of complex vision tasks such as object detection.

Motivation & Objective

• Demonstrate that a Vision Transformer backbone can be used with a detection head to perform object detection.
• Assess how transformer backbones affect detection performance and generalization on COCO.
• Investigate pretraining strategies and architectural tweaks that influence detection transfer.
• Analyze how spatial resolution, intermediate encoder features, and residual connections impact detection quality.

Proposed method

• Repurpose a Vision Transformer backbone by interpreting final transformer outputs as a spatial feature map for detection.
• Use a Faster R-CNN style detector with an RPN and RoI heads to predict class labels and bounding boxes.
• Fine-tune the entire ViT-FRCNN model end-to-end with high-resolution inputs to preserve details for small objects.
• Interpolate position embeddings to handle varying input sizes and aspect ratios during training and inference.
• Investigate architectural variants including using intermediate encoder outputs and adding residual blocks to connect the encoder to the detector.
• Pretrain backbones on large-scale image datasets (ImageNet-21k, Annotations-1.3B, Open Images) and explore curriculum pretraining.

Experimental results

Research questions

• RQ1 Can a pure Transformer backbone, when paired with a conventional detection head, achieve competitive object detection performance?
• RQ2 How do input spatial resolution and feature-map preparation affect detection accuracy, especially for small objects?
• RQ3 What is the impact of large-scale pretraining and curriculum pretraining on detection transfer performance?
• RQ4 Do intermediate encoder features and architectural connections improve detector performance when using ViT backbones?
• RQ5 How does ViT-FRCNN generalize to out-of-domain data compared to CNN-based detectors?

Key findings

• ViT-FRCNN achieves competitive COCO AP compared to ResNet-FRCNN-FPN baselines, with higher improvements when using smaller patch sizes (16x16) rather than larger (32x32).
• Reducing patch size from 32x32 to 16x16 yields substantial AP gains, particularly for small objects (AP_S).
• Incorporating intermediate encoder outputs and residual blocks boosts AP, with diminishing returns beyond a certain number of blocks.
• ViT-FRCNN shows stronger generalization on out-of-domain datasets (ObjectNet-D) and benefits from larger-scale pretraining, including Open Images V6, improving AP by up to around 2–3 points in some settings.
• Overdetections are reduced in transformer-based detectors, especially under lax NMS, indicating better suppression of spurious boxes.
• Curriculum pretraining on Open Images V6 provides additional AP gains over ImageNet-21k pretraining, particularly for small/medium objects.