[Paper Review] Class-agnostic Object Detection with Multi-modal Transformer

Muhammad Maaz, Hanoona Rasheed+4 more|arXiv (Cornell University)|Nov 22, 2021

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a class-agnostic object detection method using Multi-modal Vision Transformers (MViTs) trained on image-text pairs, leveraging natural language queries to generate high-quality object proposals across diverse domains. The key contribution is demonstrating that MViTs achieve state-of-the-art performance in generic object localization without class-specific supervision, primarily due to the structural semantics of language during training, even when captions are removed, highlighting the importance of language structure over lexical content.

ABSTRACT

What constitutes an object? This has been a long-standing question in computer vision. Towards this goal, numerous learning-free and learning-based approaches have been developed to score objectness. However, they generally do not scale well across new domains and novel objects. In this paper, we advocate that existing methods lack a top-down supervision signal governed by human-understandable semantics. For the first time in literature, we demonstrate that Multi-modal Vision Transformers (MViT) trained with aligned image-text pairs can effectively bridge this gap. Our extensive experiments across various domains and novel objects show the state-of-the-art performance of MViTs to localize generic objects in images. Based on the observation that existing MViTs do not include multi-scale feature processing and usually require longer training schedules, we develop an efficient MViT architecture using multi-scale deformable attention and late vision-language fusion. We show the significance of MViT proposals in a diverse range of applications including open-world object detection, salient and camouflage object detection, supervised and self-supervised detection tasks. Further, MViTs can adaptively generate proposals given a specific language query and thus offer enhanced interactability. Code: \url{https://git.io/J1HPY}.

Motivation & Objective

• To address the challenge of class-agnostic object detection, which lacks scalable and generalizable solutions across domains and novel objects.
• To investigate whether multi-modal vision transformers (MViTs) trained on image-text pairs can serve as effective, general-purpose object proposal generators without class-specific supervision.
• To explore the contribution of language structure versus lexical content in enabling generalization to generic object concepts in vision models.
• To develop an efficient, flexible MViT architecture (MAVL) with multi-scale deformable attention and late fusion for improved object localization.
• To demonstrate the utility of MViT-based proposals in downstream tasks such as open-world detection, salient object detection, and self-supervised learning.

Proposed method

• The method employs a Multi-modal Vision Transformer (MViT) architecture pre-trained on large-scale image-text datasets with aligned image-caption pairs.
• A novel architecture, Multi-scale Attention ViT with Late fusion (MAVL), integrates multi-scale feature extraction via deformable attention and fuses vision and language representations at a late stage to improve localization accuracy.
• Object proposals are generated by prompting the MViT with natural language queries such as 'all objects' or domain-specific terms, enabling interactive and adaptive proposal generation.
• The model is trained end-to-end using contrastive learning objectives that align visual regions with corresponding textual descriptions, even when captions are removed during training.
• Ablation studies are conducted by removing textual inputs while preserving the data loader structure to isolate the contribution of language structure to generalization.
• The framework is evaluated across multiple domains including natural images, satellite imagery, sketches, and paintings, using standard detection benchmarks.

Experimental results

Research questions

• RQ1 Can pre-trained Multi-modal Vision Transformers (MViTs) achieve state-of-the-art performance in class-agnostic object detection without any class-specific supervision?
• RQ2 What is the relative contribution of language semantics versus language structure (e.g., sequence and repetition patterns) in enabling generalization to generic object concepts?
• RQ3 How does the proposed MAVL architecture with multi-scale deformable attention and late fusion improve object localization compared to standard MViTs?
• RQ4 To what extent can MViT-generated proposals enhance performance in downstream tasks such as open-world detection, salient object detection, and self-supervised learning?
• RQ5 Can MViTs generalize to out-of-domain and novel object categories when prompted with simple natural language queries?

Key findings

• MAVL achieves state-of-the-art performance in class-agnostic object detection, outperforming both bottom-up proposal methods and standard MViTs across diverse domains including natural images, satellite imagery, sketches, and paintings.
• Even when textual captions are entirely removed during training, the model maintains strong generalization, indicating that the language structure (e.g., repeated image visits with varying contexts) is more critical than lexical content.
• Ablation studies show that preserving the language structure (e.g., multiple captions per image with varying contexts) leads to a 25.5 AP50 improvement on Pascal-VOC compared to removing all structure (from 16.2 to 61.6 AP50).
• Using only the top-30 MViT proposals in place of Selective Search in DETReg pretraining improves unsupervised object localization performance, demonstrating the quality of MViT proposals.
• For salient and camouflage object detection, task-specific text queries enable competitive performance against fully supervised models without any task-specific fine-tuning.
• The model generalizes effectively to novel object categories and domains, including sketches and paintings, due to the semantic and spatial context encoded in image-caption pairs during pretraining.