[Paper Review] Learning from Rich Semantics and Coarse Locations for Long-tailed Object Detection

Lingchen Meng, Xiyang Dai+8 more|arXiv (Cornell University)|Oct 18, 2023

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes RichSem, a simple yet effective method for long-tailed object detection that leverages rich semantics from image-text contrastive models (e.g., CLIP) and coarse image-level bounding boxes as soft supervision. By introducing a semantic branch that learns from these rich, location-robust semantics during training—without requiring precise bounding boxes—it achieves state-of-the-art performance across multiple benchmarks, especially improving detection of rare categories with minimal training overhead.

ABSTRACT

Long-tailed object detection (LTOD) aims to handle the extreme data imbalance in real-world datasets, where many tail classes have scarce instances. One popular strategy is to explore extra data with image-level labels, yet it produces limited results due to (1) semantic ambiguity -- an image-level label only captures a salient part of the image, ignoring the remaining rich semantics within the image; and (2) location sensitivity -- the label highly depends on the locations and crops of the original image, which may change after data transformations like random cropping. To remedy this, we propose RichSem, a simple but effective method, which is robust to learn rich semantics from coarse locations without the need of accurate bounding boxes. RichSem leverages rich semantics from images, which are then served as additional soft supervision for training detectors. Specifically, we add a semantic branch to our detector to learn these soft semantics and enhance feature representations for long-tailed object detection. The semantic branch is only used for training and is removed during inference. RichSem achieves consistent improvements on both overall and rare-category of LVIS under different backbones and detectors. Our method achieves state-of-the-art performance without requiring complex training and testing procedures. Moreover, we show the effectiveness of our method on other long-tailed datasets with additional experiments. Code is available at \url{https://github.com/MengLcool/RichSem}.

Motivation & Objective

• Address the long-tailed object detection problem where rare categories suffer from data scarcity.
• Overcome limitations of using image-level labels from extra datasets, which suffer from semantic ambiguity and location sensitivity.
• Improve detector performance for rare categories by leveraging rich, robust semantics from pre-trained vision-language models.
• Develop a training framework that enhances feature representation without modifying inference or requiring complex pseudo-labeling.
• Demonstrate effectiveness across multiple backbones and datasets, including LVIS and others with long-tailed distributions.

Proposed method

• Introduce a semantic branch to object detectors that learns from rich semantics extracted via CLIP using coarse image-level bounding boxes as input.
• Use the entire image as a coarse bounding box (i.e., mosaic-style input) to ensure robustness to data augmentations like random cropping.
• Extract class-level semantics from CLIP embeddings as soft labels, which serve as additional supervision during training.
• Train the detector with a unified classification loss that combines detection head loss and distillation loss from the semantic branch.
• Discard the semantic branch after training, enabling deployment as a standard detector with no inference overhead.
• Leverage large-scale, balanced classification datasets (e.g., ImageNet-21k) to enrich training data for rare classes without bounding box annotations.

Experimental results

Research questions

• RQ1 Can rich semantics from pre-trained vision-language models improve long-tailed object detection when image-level labels are used as supervision?
• RQ2 Does using coarse locations (full-image boxes) mitigate the location sensitivity issue caused by data augmentation in image-level label-based training?
• RQ3 Can soft semantics from CLIP serve as more effective supervision than one-hot image-level labels for rare category detection?
• RQ4 Does the proposed semantic branch improve feature discrimination across rare, common, and frequent categories?
• RQ5 Can the method achieve consistent performance gains across different backbones and datasets without complex training procedures?

Key findings

• RichSem achieves state-of-the-art performance on LVIS, with a 3.0 AP gain on rare categories and consistent improvements across all categories.
• The method improves detection performance by approximately 3.0 AP on rare categories even when using strong pre-trained backbones, outperforming standard pre-training.
• Feature visualization shows that RichSem produces well-separated, intra-class clustered features across all category frequencies, indicating balanced representation learning.
• The semantic branch significantly boosts performance on low-shot categories, with a 3.0 AP increase when distillation weight is set to 1, while maintaining performance on frequent categories.
• The method remains effective even when the backbone is pre-trained on large-scale data, demonstrating that it complements rather than replaces standard pre-training.
• Extensive ablation shows that both rich semantics and coarse locations are essential, with the unified loss function enabling joint optimization for detection and semantic alignment.