[Paper Review] Label, Verify, Correct: A Simple Few Shot Object Detection Method

Prannay Kaul, Weidi Xie+1 more|arXiv (Cornell University)|Dec 10, 2021

Advanced Neural Network Applications6 citations

TL;DR

This paper proposes a simple yet effective few-shot object detection method called 'Label, Verify, Correct' that improves detection performance for novel categories using only a few annotated examples. By generating high-quality pseudo-annotations through iterative label verification with a k-NN classifier and bounding box correction via a specialized regressor, the method significantly boosts recall and precision on novel classes while maintaining base category performance, achieving state-of-the-art results on PASCAL VOC and MS-COCO benchmarks.

ABSTRACT

The objective of this paper is few-shot object detection (FSOD) -- the task of expanding an object detector for a new category given only a few instances for training. We introduce a simple pseudo-labelling method to source high-quality pseudo-annotations from the training set, for each new category, vastly increasing the number of training instances and reducing class imbalance; our method finds previously unlabelled instances. Naïvely training with model predictions yields sub-optimal performance; we present two novel methods to improve the precision of the pseudo-labelling process: first, we introduce a verification technique to remove candidate detections with incorrect class labels; second, we train a specialised model to correct poor quality bounding boxes. After these two novel steps, we obtain a large set of high-quality pseudo-annotations that allow our final detector to be trained end-to-end. Additionally, we demonstrate our method maintains base class performance, and the utility of simple augmentations in FSOD. While benchmarking on PASCAL VOC and MS-COCO, our method achieves state-of-the-art or second-best performance compared to existing approaches across all number of shots.

Motivation & Objective

• To address the challenge of few-shot object detection (FSOD), where models must detect new object categories with only 1–30 annotated examples, while preserving performance on base categories.
• To overcome the limitations of supervision collapse and severe class imbalance in few-shot training, where novel classes are vastly underrepresented.
• To improve the quality of pseudo-annotations for novel categories by refining both class labels and bounding box coordinates through dedicated verification and correction steps.
• To demonstrate the effectiveness of aggressive data augmentation in few-shot object detection, a factor previously underexplored in prior work.
• To achieve state-of-the-art or second-best performance on both PASCAL VOC and MS-COCO benchmarks across all shot settings (1–30 shots).

Proposed method

• The method begins with a baseline Faster R-CNN detector trained on base categories, which is then used to generate noisy candidate detections for novel categories across the full training set.
• A k-NN classifier, trained on self-supervised features from the few-shot annotations, verifies the class labels of these candidates, filtering out false positives with mismatched predictions.
• A specialized box regressor is trained to refine the bounding box coordinates of verified candidates, significantly improving localization accuracy.
• The resulting high-precision pseudo-annotations are combined with original ground-truth annotations for base categories to retrain the detector end-to-end.
• The approach leverages aggressive data augmentation to further improve generalization and reduce overfitting on limited novel category data.
• The entire pipeline—labeling, verification, and correction—is designed to be simple and compatible with standard two-stage detectors like Faster R-CNN.

Experimental results

Research questions

• RQ1 Can pseudo-labelling with high-precision annotations significantly improve few-shot object detection performance on novel categories?
• RQ2 How can class label errors in pseudo-annotations be effectively reduced without relying on human-annotated data?
• RQ3 Can a specialized box correction module improve the quality of bounding box predictions for pseudo-annotated instances?
• RQ4 To what extent do aggressive data augmentation strategies enhance few-shot generalization in object detection?
• RQ5 Does the proposed method maintain performance on base categories while improving detection for novel classes?

Key findings

• On the MS-COCO benchmark with 30 shots, the proposed method achieved a 57.6 mAP (mean average precision) on novel classes, significantly outperforming the baseline of 44.2 mAP under the same setting.
• On PASCAL VOC, the method achieved 59.6 mAP on novel classes with 30 shots using ResNet-101, ranking as state-of-the-art or second-best across all shot settings.
• The precision-recall curves under IoU=0.75 showed a substantial improvement in both precision and recall for novel classes after applying the pseudo-labelling pipeline.
• The ablation study confirmed that both label verification and box correction are essential components, with performance dropping significantly when either step is removed.
• The method maintained strong performance on base categories, demonstrating no degradation due to supervision collapse during end-to-end retraining.
• Aggressive data augmentation was found to be critical for generalization, with ablation studies showing a clear performance gain when used in conjunction with pseudo-labelling.