[Paper Review] You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection

Y.K. Fang, Bencheng Liao+6 more|arXiv (Cornell University)|Jun 1, 2021

Advanced Neural Network ApplicationsComputer Science72 references195 citations

TL;DR

YOLOS shows a vanilla Vision Transformer with minimal modifications can perform 2D object detection using a pure sequence-to-sequence approach, achieving competitive COCO results (e.g., YOLOS-Base 42.0 AP on COCO val) after ImageNet-1k pre-training.

ABSTRACT

Can Transformer perform 2D object- and region-level recognition from a pure sequence-to-sequence perspective with minimal knowledge about the 2D spatial structure? To answer this question, we present You Only Look at One Sequence (YOLOS), a series of object detection models based on the vanilla Vision Transformer with the fewest possible modifications, region priors, as well as inductive biases of the target task. We find that YOLOS pre-trained on the mid-sized ImageNet-1k dataset only can already achieve quite competitive performance on the challenging COCO object detection benchmark, e.g., YOLOS-Base directly adopted from BERT-Base architecture can obtain 42.0 box AP on COCO val. We also discuss the impacts as well as limitations of current pre-train schemes and model scaling strategies for Transformer in vision through YOLOS. Code and pre-trained models are available at https://github.com/hustvl/YOLOS.

Motivation & Objective

• Demonstrate that a vanilla Vision Transformer can transfer from image-level recognition to 2D object detection with minimal spatial inductive biases.
• Show that object detection can be performed in a pure sequence-to-sequence framework without reinterpreting ViT outputs as 2D feature maps.
• Evaluate the impact of pre-training schemes on ViT-based detectors and establish YOLOS as a benchmark for ViT pre-training strategies.

Proposed method

• Replace the ViT CLS token with 100 DET tokens for object detection.
• Use a bipartite matching loss to train detections in a set-p prediction manner, avoiding explicit 2D spatial priors during label assignment.
• Keep the YOLOS architecture as close as possible to vanilla ViT, with only necessary detector heads implemented as lightweight MLPs for class and bbox prediction.
• Append DET tokens to PATCH tokens and add 1D learnable position embeddings; train with standard Transformer encoder layers.
• Fine-tune from ImageNet-1k pre-trained weights, except for the DET tokens and small heads which are randomly initialized.

Experimental results

Research questions

• RQ1 Can a pre-trained vanilla ViT transfer to robust 2D object detection without heavy 2D inductive biases?
• RQ2 How do different pre-training strategies (supervised vs self-supervised) affect ViT-based detectors on COCO?
• RQ3 What is the impact of model scaling on pre-training and transfer learning for ViT in object detection?
• RQ4 How does YOLOS compare to DETR and CNN-based detectors when evaluated under similar settings?

Key findings

• A vanilla ViT pre-trained on ImageNet-1k can achieve competitive COCO detection performance with minimal modifications (e.g., YOLOS-Base achieves 42.0 AP on COCO val).
• YOLOS uses 100 DET tokens as object proxies and bipartite matching loss, avoiding re-interpretation of ViT outputs into 2D feature maps.
• Pre-training strategy significantly influences transfer to COCO; self-supervised pre-training (e.g., DINO) can approach label-supervised performance for smaller models, and distillation benefits exist.
• Among scaling strategies, width scaling, uniform compound scaling (dwr), and fast scaling (dw r) show different trade-offs; pre-training gains do not always translate directly to COCO due to the prominence of spatial attention at higher resolutions.
• YOLOS-Ti is competitive with tiny CNN detectors, and larger YOLOS variants can outperform some DETR counterparts in certain settings, though gains are model- and setting-dependent.
• The DET tokens mainly carry information for object locations rather than object categories, indicating their role as position-aware object proxies rather than category-specific embeddings.