[Paper Review] Hallucination Improves Few-Shot Object Detection

Weilin Zhang, Yu-Xiong Wang|arXiv (Cornell University)|May 4, 2021

Domain Adaptation and Few-Shot Learning46 references11 citations

TL;DR

This paper proposes a hallucinator network that generates synthetic region-of-interest (RoI) features to augment few-shot object detection training, transferring within-class variation from base classes to improve classifier generalization. By iteratively hallucinating diverse, realistic examples in feature space, the method achieves state-of-the-art performance on COCO and PASCAL VOC in the extremely-few-shot regime (1–2 shots), significantly reducing false positives and improving detection accuracy.

ABSTRACT

Learning to detect novel objects from few annotated examples is of great practical importance. A particularly challenging yet common regime occurs when there are extremely limited examples (less than three). One critical factor in improving few-shot detection is to address the lack of variation in training data. We propose to build a better model of variation for novel classes by transferring the shared within-class variation from base classes. To this end, we introduce a hallucinator network that learns to generate additional, useful training examples in the region of interest (RoI) feature space, and incorporate it into a modern object detection model. Our approach yields significant performance improvements on two state-of-the-art few-shot detectors with different proposal generation procedures. In particular, we achieve new state of the art in the extremely-few-shot regime on the challenging COCO benchmark.

Motivation & Objective

• Address the critical challenge of limited label variation in few-shot object detection, especially when only 1–2 examples are available per novel class.
• Overcome the limitations of existing methods that rely solely on region proposal networks (RPNs) or meta-learning, which fail to capture sufficient appearance variation.
• Improve classifier robustness by transferring shared within-class variation (e.g., pose, lighting) from data-abundant base classes to novel classes.
• Develop a plug-and-play hallucination module compatible with modern two-stage detectors like TFA and CoRPNs, without requiring complex meta-learning.

Proposed method

• Train a hallucinator network in an EM-like manner: first generate synthetic RoI features using the current classifier, then re-train the classifier on real and hallucinated examples.
• Perform hallucination in the learned RoI feature space, enabling the model to generate diverse, realistic variations of novel class features.
• Use a simple, end-to-end training procedure that alternates between classifier refinement and hallucination, avoiding the need for meta-learning.
• Generate a fixed number of hallucinated examples per class per training batch (e.g., 20), with the number tuned for optimal performance.
• Incorporate the hallucinated examples into the classifier’s training loss, improving decision boundary estimation.
• Apply the hallucinator during inference fine-tuning, enabling better generalization even with minimal real examples.

Experimental results

Research questions

• RQ1 Can data hallucination in feature space effectively address the lack of appearance variation in extremely-few-shot object detection?
• RQ2 How does hallucinating examples in RoI feature space compare to relying only on RPN-generated proposals for training classifiers?
• RQ3 Does the proposed EM-like training procedure for the hallucinator yield better performance than meta-learning-based alternatives in few-shot detection?
• RQ4 To what extent does hallucination reduce false positive predictions by improving decision boundary estimation?
• RQ5 Can the hallucinator be effectively integrated into different state-of-the-art few-shot detectors without architectural overhaul?

Key findings

• The proposed hallucinator improves 1-shot AP50 by 5.3 points on PASCAL VOC novel set 1 over TFA, achieving 45.1 compared to TFA’s 39.8.
• On novel set 2, the method achieves 29.3 AP50 with the aggressive hallucinator, a 6.1-point improvement over TFA’s 23.5.
• The hallucinator reduces false positive boxes by up to 25% (from 3,440.6 to 3,041.1) on average, indicating better decision boundary estimation.
• Performance saturates at 20 hallucinated examples per batch, with a slight drop beyond that, suggesting optimal capacity.
• The method improves both true positive detection and false positive suppression, with the latter showing larger gains, indicating improved boundary generalization.
• The hallucinator is effective across different classifier types: it improves both cosine and fully-connected classifiers in CoRPNs, confirming its generalizability.