[Paper Review] FreeAnchor: Learning to Match Anchors for Visual Object Detection

Xiaosong Zhang, Fang Wan+3 more|arXiv (Cornell University)|Sep 5, 2019

Advanced Neural Network Applications275 citations

TL;DR

FreeAnchor reframes detector training as a maximum likelihood estimation process to learn flexible anchor matching from a bag of anchors, improving detection performance over IoU-based assignment.

ABSTRACT

Modern CNN-based object detectors assign anchors for ground-truth objects under the restriction of object-anchor Intersection-over-Unit (IoU). In this study, we propose a learning-to-match approach to break IoU restriction, allowing objects to match anchors in a flexible manner. Our approach, referred to as FreeAnchor, updates hand-crafted anchor assignment to "free" anchor matching by formulating detector training as a maximum likelihood estimation (MLE) procedure. FreeAnchor targets at learning features which best explain a class of objects in terms of both classification and localization. FreeAnchor is implemented by optimizing detection customized likelihood and can be fused with CNN-based detectors in a plug-and-play manner. Experiments on COCO demonstrate that FreeAnchor consistently outperforms their counterparts with significant margins.

Motivation & Objective

• Motivate breaking the hand-crafted IoU-based anchor assignment in anchor-based detectors.
• Propose a learning-to-match framework to select anchors for objects.
• Ensure recall, precision, and NMS compatibility through a detection customized likelihood.
• Enable end-to-end optimization of classification, localization, and anchor matching.

Proposed method

• Formulate detector training as maximum likelihood estimation (MLE) over object-anchor pairs.
• Define detection customized likelihood to jointly optimize recall and precision with NMS compatibility.
• Construct anchor bags for each object by selecting top-n anchors by IoU and learn to pick the best from each bag.
• Introduce a Mean-max anchor selection mechanism that evolves from mean-like to max-like as training proceeds.
• Replace RetinaNet loss with a detection customized loss (Eq. 7) incorporating balanced terms and focal loss.
• Integrate the approach in a plug-and-play manner with existing CNN-based detectors.

Experimental results

Research questions

• RQ1 Can learning to match anchors from a candidate bag improve object detection beyond hand-crafted IoU-based assignment?
• RQ2 Does the detection customized likelihood enable better recall, precision, and NMS compatibility?
• RQ3 How does the Mean-max anchor selection strategy affect training dynamics and final accuracy?

Key findings

• FreeAnchor consistently improves AP over the RetinaNet baseline on COCO.
• With the same backbone, FreeAnchor achieves up to about 3.0 AP point gains over RetinaNet.
• FreeAnchor shows higher NMS recall across IoU thresholds, indicating better compatibility with NMS.
• FreeAnchor outperforms several state-of-the-art one-stage detectors and anchor-free methods on COCO benchmarks.
• Anchor matching becomes more effective in crowded scenes and for slender objects due to learned anchor selection.
• The method maintains training efficiency with a modest increase in training time.