[Paper Review] RepPoints: Point Set Representation for Object Detection

Ze Yang, Shaohui Liu+3 more|arXiv (Cornell University)|Apr 25, 2019

Advanced Neural Network ApplicationsComputer Science55 references146 citations

TL;DR

RepPoints introduces a learnable set of representative sample points to model objects, enabling anchor-free detectors (RPDet) that match or exceed anchor-based methods on COCO, with strong multi-scale results.

ABSTRACT

Modern object detectors rely heavily on rectangular bounding boxes, such as anchors, proposals and the final predictions, to represent objects at various recognition stages. The bounding box is convenient to use but provides only a coarse localization of objects and leads to a correspondingly coarse extraction of object features. In this paper, we present extbf{RepPoints} (representative points), a new finer representation of objects as a set of sample points useful for both localization and recognition. Given ground truth localization and recognition targets for training, RepPoints learn to automatically arrange themselves in a manner that bounds the spatial extent of an object and indicates semantically significant local areas. They furthermore do not require the use of anchors to sample a space of bounding boxes. We show that an anchor-free object detector based on RepPoints can be as effective as the state-of-the-art anchor-based detection methods, with 46.5 AP and 67.4 $AP_{50}$ on the COCO test-dev detection benchmark, using ResNet-101 model. Code is available at https://github.com/microsoft/RepPoints.

Motivation & Objective

• Motivate the need for finer object localization beyond rectangular bounding boxes.
• Propose RepPoints as a learnable set of sample points that bound an object and highlight semantically important regions.
• Develop an anchor-free detector (RPDet) that uses RepPoints for initialization, refinement, and final localization.
• Demonstrate that RepPoints can match or surpass anchor-based detectors on COCO with and without multi-scale training/testing.

Proposed method

• Represent objects as a differentiable set of n sample points, RepPoints, instead of a 4-d bounding box.
• Refine RepPoints across stages via predicted offsets to tighten object localization.
• Convert RepPoints to a pseudo box with differentiable transformations for supervision and evaluation.
• Train with a combination of localization loss (pseudo box distance to ground truth) and recognition loss from the next detection stage.
• Integrate RepPoints with deformable convolutions to enable sampling at irregular points.
• Use an anchor-free center-point initiation that evolves into RepPoints proposals and final targets.

Experimental results

Research questions

• RQ1 Can RepPoints provide finer localization than bounding boxes while maintaining or improving recognition performance?
• RQ2 Is an anchor-free detector using RepPoints (RPDet) competitive with state-of-the-art anchor-based detectors on COCO?
• RQ3 How does RepPoints learning supervision (localization + recognition) influence the placement of semantically meaningful points?
• RQ4 What is the impact of transforming RepPoints to pseudo boxes for training and evaluation?

Key findings

• RPDet with RepPoints achieves 42.8 AP on COCO test-dev, on par with Cascade R-CNN and better than existing anchor-free detectors.
• With multi-scale training and testing, RPDet reaches 46.5 AP and 67.4 AP50 on COCO test-dev, surpassing prior detectors.
• Replacing bounding boxes with RepPoints yields improvements over a bounding-box baseline (36.2 AP vs 38.3 AP with ResNet-50).
• Center-point initialization (a special RepPoints case) provides better initial object hypotheses than anchors, contributing to higher performance.
• RepPoints learned points concentrate on extreme or semantic keypoints, enabling finer localization without extra supervision.
• Applying deformable RoI pooling to RepPoints proposals yields additional gains, indicating complementary use with deformable sampling.
• Various differentiable transforming functions from RepPoints to pseudo boxes perform comparably, enabling end-to-end training.