[Paper Review] Prime Sample Attention in Object Detection

Yuhang Cao, Kai Chen+2 more|arXiv (Cornell University)|Apr 9, 2019

Advanced Neural Network Applications35 references44 citations

TL;DR

The paper introduces Prime Sample Attention (PISA), a sampling and learning strategy that focuses training on prime samples (highly influential positive/negative proposals) using Hierarchical Local Rank (HLR) and a classification-aware regression loss to improve mAP, outperforming random sampling and hard mining on COCO and VOC benchmarks.

ABSTRACT

It is a common paradigm in object detection frameworks to treat all samples equally and target at maximizing the performance on average. In this work, we revisit this paradigm through a careful study on how different samples contribute to the overall performance measured in terms of mAP. Our study suggests that the samples in each mini-batch are neither independent nor equally important, and therefore a better classifier on average does not necessarily mean higher mAP. Motivated by this study, we propose the notion of Prime Samples, those that play a key role in driving the detection performance. We further develop a simple yet effective sampling and learning strategy called PrIme Sample Attention (PISA) that directs the focus of the training process towards such samples. Our experiments demonstrate that it is often more effective to focus on prime samples than hard samples when training a detector. Particularly, On the MSCOCO dataset, PISA outperforms the random sampling baseline and hard mining schemes, e.g., OHEM and Focal Loss, consistently by around 2% on both single-stage and two-stage detectors, even with a strong backbone ResNeXt-101.

Motivation & Objective

• Question the assumption that all mini-batch samples contribute equally to mAP.
• Identify which samples most influence detection performance and how to rank them.
• Propose a practical sampling and loss strategy that emphasizes prime samples during training.
• Demonstrate improvements across both two-stage and single-stage detectors on COCO and VOC.

Proposed method

• Define Prime Samples as samples most influential to detection performance.
• Introduce Hierarchical Local Rank (HLR) to rank positive samples by IoU and negative samples by score within mini-batches.
• Develop an Importance-based Sample Reweighting (ISR) to convert HLR ranks into loss weights for positives and negatives.
• Propose Classification-Aware Regression Loss (CARL) to jointly optimize classification and regression with sample-aware weighting.
• Apply PISA to both two-stage and single-stage detectors without adding inference overhead.
• Show that PISA yields gains over random sampling and hard mining on COCO and VOC.

Experimental results

Research questions

• RQ1 What samples are most important for training an object detector, and how can we quantify their importance?
• RQ2 Does prioritizing prime samples during training improve mAP more than traditional random sampling or hard mining?
• RQ3 How can we jointly optimize classification and localization to reinforce attention to prime samples?

Key findings

• PISA consistently improves mAP across Faster R-CNN, Mask R-CNN, RetinaNet, and SSD-based detectors on COCO with backbones like ResNet-50 and ResNeXt-101-32x4d.
• On COCO test-dev, PISA yields about 2% absolute mAP gains over baselines for both single-stage and two-stage detectors.
• PISA performs better than random sampling and hard mining for positive and negative samples, with notable gains at higher IoU thresholds (e.g., AP75).
• HLR-based ranking places high-IoU positive samples near the top and high-score negatives at the top of their respective ranked lists, guiding training to prime samples.
• CARL correlates classification and regression by using regression loss to modulate classification scores, boosting prime samples.
• PISA achieves improvements on VOC07 as well, indicating generality across datasets.