[Paper Review] Frustratingly Simple but Effective Zero-shot Detection and Segmentation: Analysis and a Strong Baseline

Siddhesh Khandelwal, Anirudth Nambirajan+3 more|arXiv (Cornell University)|Feb 14, 2023

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a simple yet highly effective zero-shot object detection and instance segmentation method that outperforms complex state-of-the-art models by leveraging pre-trained visual-linguistic embeddings (e.g., CLIP) and a two-stage training process. The approach achieves state-of-the-art performance on MSCOCO, with up to 37.3% higher mAP and 19.6% higher recall than prior methods on challenging zero-shot splits.

ABSTRACT

Methods for object detection and segmentation often require abundant instance-level annotations for training, which are time-consuming and expensive to collect. To address this, the task of zero-shot object detection (or segmentation) aims at learning effective methods for identifying and localizing object instances for the categories that have no supervision available. Constructing architectures for these tasks requires choosing from a myriad of design options, ranging from the form of the class encoding used to transfer information from seen to unseen categories, to the nature of the function being optimized for learning. In this work, we extensively study these design choices, and carefully construct a simple yet extremely effective zero-shot recognition method. Through extensive experiments on the MSCOCO dataset on object detection and segmentation, we highlight that our proposed method outperforms existing, considerably more complex, architectures. Our findings and method, which we propose as a competitive future baseline, point towards the need to revisit some of the recent design trends in zero-shot detection / segmentation.

Motivation & Objective

• To address the high cost and scarcity of instance-level annotations in object detection and segmentation by enabling zero-shot generalization to unseen categories.
• To investigate the impact of key design choices—such as embedding type, loss function, and model architecture—on zero-shot performance.
• To challenge the prevailing trend of designing complex architectures by demonstrating that simplicity with careful ablation can yield superior results.
• To establish a strong, reproducible baseline for future research in zero-shot detection and segmentation.
• To highlight the underexplored potential of high-quality pre-trained semantic embeddings like CLIP for zero-shot transfer.

Proposed method

• The method uses a two-stage training process: first pre-training a Faster R-CNN or Mask R-CNN detector on seen categories with bounding box and mask annotations.
• In the second stage, a projection head is fine-tuned using cross-entropy loss to map image features to a semantic embedding space.
• Semantic embeddings for unseen categories are derived from pre-trained models like CLIP or GloVe, enabling zero-shot transfer without additional supervision.
• The model leverages normalized class-name embeddings to compute classification scores during inference, enabling detection and segmentation of unseen classes.
• The approach uses standard cross-entropy loss during fine-tuning, avoiding complex auxiliary losses or attention mechanisms.
• The final inference combines region proposals with semantic embedding similarity to predict both seen and unseen category instances.

Figure 2 : Proposed Approach with Two-step Training. The first step trains the learnable parameters of a Faster [ 37 ] / Mask [ 13 ] RCNN architecture using the seen category annotations. The second step freezes these parameters, and learns the projection matrices which, along with the semantic embe

Experimental results

Research questions

• RQ1 How do different choices of semantic embeddings (e.g., Word2Vec vs. CLIP) affect zero-shot detection and segmentation performance?
• RQ2 Does a simpler model architecture with minimal design complexity outperform more complex state-of-the-art models in zero-shot detection and segmentation?
• RQ3 What is the relative impact of loss function, model capacity, and inference strategy on zero-shot generalization?
• RQ4 Can a two-stage training process with fine-tuned projection heads achieve state-of-the-art performance without architectural complexity?
• RQ5 To what extent does the choice of embedding source dominate performance compared to architectural innovations?

Key findings

• The proposed method with CLIP embeddings (F+CLIP) achieves a 37.3% improvement in mAP and 13.1% increase in recall over the prior state-of-the-art on the 48/17 split in zero-shot detection.
• On the more challenging generalized zero-shot detection (GZSD) setup, the F+CLIP variant achieves an average 11% improvement in harmonic mean mAP over the closest baseline.
• For zero-shot instance segmentation, the method outperforms the prior state-of-the-art by 117.4% on HM mAP and 12.8% on HM recall across both splits.
• The use of CLIP embeddings leads to significantly better performance than Word2Vec, highlighting the importance of high-quality visio-linguistic pre-training.
• Despite its simplicity, the method surpasses more complex architectures, suggesting that architectural complexity is not a prerequisite for strong zero-shot generalization.
• Ablation studies show that the choice of semantic embedding has a larger impact on performance than architectural design choices, a factor often overlooked in prior work.