[Paper Review] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

Shilong Liu, Zhaoyang Zeng+10 more|arXiv (Cornell University)|Mar 9, 2023

Multimodal Machine Learning Applications240 citations

TL;DR

Grounding DINO extends DINO with grounded pre-training to enable open-set object detection and referring expression comprehension by tightly fusing language and vision across multiple pipeline phases.

ABSTRACT

In this paper, we present an open-set object detector, called Grounding DINO, by marrying Transformer-based detector DINO with grounded pre-training, which can detect arbitrary objects with human inputs such as category names or referring expressions. The key solution of open-set object detection is introducing language to a closed-set detector for open-set concept generalization. To effectively fuse language and vision modalities, we conceptually divide a closed-set detector into three phases and propose a tight fusion solution, which includes a feature enhancer, a language-guided query selection, and a cross-modality decoder for cross-modality fusion. While previous works mainly evaluate open-set object detection on novel categories, we propose to also perform evaluations on referring expression comprehension for objects specified with attributes. Grounding DINO performs remarkably well on all three settings, including benchmarks on COCO, LVIS, ODinW, and RefCOCO/+/g. Grounding DINO achieves a $52.5$ AP on the COCO detection zero-shot transfer benchmark, i.e., without any training data from COCO. It sets a new record on the ODinW zero-shot benchmark with a mean $26.1$ AP. Code will be available at \url{https://github.com/IDEA-Research/GroundingDINO}.

Motivation & Objective

• Motivate open-set object detection by enabling detection of arbitrary objects described by language inputs.
• Extend a closed-set detector with language to generalize to novel concepts.
• Propose a multi-phase fusion approach to tightly integrate vision and language in a Transformer detector.

Proposed method

• Introduce a feature enhancer that stacks self-attention and image-text cross-attention for cross-modality fusion in the neck.
• Implement a language-guided query selection module to initialize decoder queries from image and text features.
• Add a cross-modality decoder with image and text cross-attention layers for joint refinement of queries.
• Represent text prompts at sub-sentence level to avoid unwanted interactions among category names.
• Use a contrastive loss between predicted objects and language tokens for classification.
• Train in a dual-encoder-single-decoder architecture leveraging a Swin Transformer image backbone and a BERT text backbone.

Experimental results

Research questions

• RQ1 How can language information be integrated into a closed-set detector to enable open-set object detection?
• RQ2 Does tight fusion across neck, queries, and decoder improve open-set and REC performance compared to partial fusion approaches?
• RQ3 Can Grounding DINO achieve strong zero-shot and ODinW results and extend to referring expression comprehension datasets?
• RQ4 What is the impact of sub-sentence level text representations on multi-category grounding performance?

Key findings

• Grounding DINO achieves 52.5 AP on COCO zero-shot transfer without COCO training data and 63.0 AP after COCO fine-tuning.
• Grounding DINO sets a new state of the art on the ODinW zero-shot benchmark with a mean AP of 26.1.
• Grounding DINO outperforms GLIP on LVIS and ODinW under comparable settings and shows stronger scalability with more data.
• The model extends open-set evaluation to RefCOCO/+/g datasets, illustrating REC capability.
• A transfer-from-DINO ablation shows comparable or improved performance when initializing Grounding DINO from pre-trained DINO weights.