[Paper Review] Frustratingly Simple Few-Shot Object Detection

Xin Wang, Thomas E. Huang+3 more|arXiv (Cornell University)|Mar 16, 2020

Domain Adaptation and Few-Shot Learning22 references40 citations

TL;DR

The paper shows that fine-tuning only the last layer of an object detector on a balanced set of base and novel classes (two-stage fine-tuning) plus instance-level feature normalization and cosine-similarity classification outperforms meta-learning methods for few-shot object detection, and introduces more reliable benchmarks across VOC, COCO, and LVIS.

ABSTRACT

Detecting rare objects from a few examples is an emerging problem. Prior works show meta-learning is a promising approach. But, fine-tuning techniques have drawn scant attention. We find that fine-tuning only the last layer of existing detectors on rare classes is crucial to the few-shot object detection task. Such a simple approach outperforms the meta-learning methods by roughly 2~20 points on current benchmarks and sometimes even doubles the accuracy of the prior methods. However, the high variance in the few samples often leads to the unreliability of existing benchmarks. We revise the evaluation protocols by sampling multiple groups of training examples to obtain stable comparisons and build new benchmarks based on three datasets: PASCAL VOC, COCO and LVIS. Again, our fine-tuning approach establishes a new state of the art on the revised benchmarks. The code as well as the pretrained models are available at https://github.com/ucbdrive/few-shot-object-detection.

Motivation & Objective

• Assess whether simple fine-tuning can outperform meta-learning for few-shot object detection.
• Investigate the impact of training schedule and instance-level feature normalization on detection performance.
• Revise evaluation protocols to address variability and base-class knowledge retention in few-shot settings.

Proposed method

• Adopt Faster R-CNN as the base detector and split training into base and few-shot stages (two-stage fine-tuning, TFA).
• In stage one, train the detector on base classes; in stage two, freeze feature extractors and fine-tune only the last layers of the box predictor on a balanced set of base and novel classes.
• In the second stage, apply instance-level feature normalization within the box classifier to reduce intra-class variance.
• Evaluate using a cosine-similarity based box classifier to classify proposals, with a fixed scaling factor alpha (empirically 20).
• Compare with meta-learning baselines (FSRW, Meta R-CNN, MetaDet) and with fully fine-tuned or jointly trained baselines.
• Revise benchmarks by sampling multiple training groups, reporting base-class AP, novel-class AP, and generalized AP on VOC, COCO, and LVIS.

Experimental results

Research questions

• RQ1 Does fine-tuning only the last layers of a detector on a balanced base+novel set outperform meta-learning approaches in few-shot object detection?
• RQ2 What training mechanics (schedule, normalization, cosine classifier) best stabilize and improve few-shot detection performance?
• RQ3 How do revised evaluation protocols affect reliability and comparisons across datasets (VOC, COCO, LVIS) in few-shot detection?

Key findings

• Two-stage fine-tuning (TFA) consistently outperforms prior meta-learning methods by about 2 to 20 AP points on VOC and COCO benchmarks.
• Using a cosine-similarity based box classifier with instance-level normalization further improves novel-class AP, especially in very low-shot regimes (1-shot).
• On LVIS, TFA improves rare-class AP by ~4 points and common-class AP by ~2 points with negligible impact on frequent classes.
• The revised benchmarks reveal high variance in few-shot detection results; multiple runs with different sampling groups yield more stable and reliable comparisons.
• Across VOC, COCO, and LVIS, TFA establishes new state-of-the-art results under the revised evaluation protocols.