[Paper Review] Pix2seq: A Language Modeling Framework for Object Detection

Ting Chen, Saurabh Saxena+3 more|arXiv (Cornell University)|Sep 22, 2021

Multimodal Machine Learning Applications58 references128 citations

TL;DR

Pix2Seq reframes object detection as a language modeling task that generates sequences of discrete tokens representing bounding boxes and class labels from image inputs, achieving competitive COCO results with a simple, generic architecture.

ABSTRACT

We present Pix2Seq, a simple and generic framework for object detection. Unlike existing approaches that explicitly integrate prior knowledge about the task, we cast object detection as a language modeling task conditioned on the observed pixel inputs. Object descriptions (e.g., bounding boxes and class labels) are expressed as sequences of discrete tokens, and we train a neural network to perceive the image and generate the desired sequence. Our approach is based mainly on the intuition that if a neural network knows about where and what the objects are, we just need to teach it how to read them out. Beyond the use of task-specific data augmentations, our approach makes minimal assumptions about the task, yet it achieves competitive results on the challenging COCO dataset, compared to highly specialized and well optimized detection algorithms.

Motivation & Objective

• Motivate a generic, task-agnostic approach to object detection that minimizes hand-crafted architecture and loss design.
• Represent object descriptions (bounding boxes and class labels) as sequences of discrete tokens.
• Demonstrate that an encoder–decoder model trained with maximum likelihood can detect objects from pixel inputs without task-specific priors.
• Show that sequence augmentation and token-based quantization enable competitive performance on COCO, with benefits from pretraining on larger datasets.

Proposed method

• Quantize bounding boxes into five-token sequences [y_min, x_min, y_max, x_max, c] using a shared vocabulary of bins plus class tokens.
• Serialize multiple object descriptions into a single sequence with a random object ordering and an EOS token.
• Use an encoder (image percept) + Transformer decoder to generate tokens autoregressively and maximize the token likelihood.
• Train with a simple softmax cross-entropy loss on tokens conditioned on the image and preceding tokens (maximum likelihood).
• Apply sequence augmentation by injecting synthetic noise objects into input sequences and labeling them as noise in the target to improve recall and robustness.
• During inference, sample tokens (e.g., nucleus sampling) until EOS is produced; extract and de-quantize bounding boxes and labels from the generated token sequence.

Experimental results

Research questions

• RQ1 Can object detection be effectively cast as a language modeling problem conditioned on image inputs?
• RQ2 Does a generic encoder–decoder with token-based bounding box and class representations achieve competitive COCO results without task-specific object proposals or losses?
• RQ3 What is the impact of sequence augmentation and object-order randomness on detection recall and precision?
• RQ4 How does pretraining on a larger detection dataset affect performance on COCO?
• RQ5 What are the trade-offs in using token-based quantization for bounding boxes in terms of precision and model capacity?

Key findings

• Pix2Seq achieves competitive results to Faster R-CNN and DETR on COCO across multiple backbones and settings.
• With ResNet-50, Pix2Seq attains AP 43.0 and AP75 45.6, outperforming some baselines in certain metrics; with ResNet-101, AP 44.5 and AP75 47.5 are observed.
• Pretraining on Objects365 followed by fine-tuning on COCO yields strong gains, e.g., ViT-L reaches AP 50.0 at 1333x1333 during finetuning (best among listed finetuned configurations).
• Quantization with 500+ bins provides high-precision bounding boxes (1.3 pixels per bin at 640 longest side) without notable loss, and random object ordering during training improves AP and AR compared with deterministic orderings.
• Sequence augmentation significantly improves recall (especially when fine-tuning) with modest or minor AP impact; delaying EOS via likelihood offsetting helps recall.
• Among inference strategies, nucleus sampling improves recall over argmax.