[Paper Review] VarifocalNet: An IoU-aware Dense Object Detector

Haoyang Zhang, Ying Wang+2 more|arXiv (Cornell University)|Aug 31, 2020

Advanced Neural Network ApplicationsComputer Science51 references32 citations

TL;DR

VFNet introduces IoU-aware classification scores (IACS) and trains with Varifocal Loss to jointly model object presence and localization, improving dense detector ranking and achieving state-of-the-art COCO results.

ABSTRACT

Accurately ranking the vast number of candidate detections is crucial for dense object detectors to achieve high performance. Prior work uses the classification score or a combination of classification and predicted localization scores to rank candidates. However, neither option results in a reliable ranking, thus degrading detection performance. In this paper, we propose to learn an Iou-aware Classification Score (IACS) as a joint representation of object presence confidence and localization accuracy. We show that dense object detectors can achieve a more accurate ranking of candidate detections based on the IACS. We design a new loss function, named Varifocal Loss, to train a dense object detector to predict the IACS, and propose a new star-shaped bounding box feature representation for IACS prediction and bounding box refinement. Combining these two new components and a bounding box refinement branch, we build an IoU-aware dense object detector based on the FCOS+ATSS architecture, that we call VarifocalNet or VFNet for short. Extensive experiments on MS COCO show that our VFNet consistently surpasses the strong baseline by $\sim$2.0 AP with different backbones. Our best model VFNet-X-1200 with Res2Net-101-DCN achieves a single-model single-scale AP of 55.1 on COCO test-dev, which is state-of-the-art among various object detectors.Code is available at https://github.com/hyz-xmaster/VarifocalNet .

Motivation & Objective

• Motivate accurate ranking of dense detector candidate boxes beyond plain classification scores.
• Propose an IoU-aware classification score (IACS) that fuses object presence with localization accuracy.
• Develop a new loss (Varifocal Loss) to train detectors to predict IACS with asymmetric weighting.
• Introduce a star-shaped bounding box feature representation for IACS prediction and box refinement.
• Integrate these components into an FCOS+ATSS-based architecture to form VarifocalNet (VFNet) and evaluate on COCO.

Proposed method

• Define IACS as a scalar where the ground-truth class position holds the IoU between predicted and ground-truth boxes, and other positions are zero.
• Introduce Varifocal Loss (VFL), a dynamically scaled binary cross-entropy loss that down-weights negatives asymmetrically and up-weights high-quality positives using the target q (gt_IoU for positives).
• Adopt a star-shaped bounding box representation using nine fixed sampling points with deformable convolution to encode box geometry and context for IACS prediction.
• Add a bounding box refinement branch that predicts distance-scaling factors to refine the initial box via a residual formulation.
• Remove the centerness branch from FCOS+ATSS and build VFNet with two heads: one for IACS prediction (classification) and one for bounding box localization and refinement.

Experimental results

Research questions

• RQ1 Can an IoU-aware classification score (IACS) provide a more reliable ranking basis for dense detectors than traditional classification or multiplied localization signals?
• RQ2 Does Varifocal Loss improve training dynamics and final detection performance for predicting IACS?
• RQ3 Does a star-shaped fixed-point feature representation better capture box geometry and context for IACS prediction and refinement?
• RQ4 How does VFNet compare to strong baselines (e.g., FCOS+ATSS) on COCO in terms of AP and efficiency?
• RQ5 Is it feasible to achieve state-of-the-art single-model COCO results with aVFNet-X variant using advanced backbones and training schemes?

Key findings

• IACS provides a superior ranking basis for detections over traditional scores or products of signals.
• Varifocal Loss improves AP over FCOS+ATSS and enables effective learning of continuous IACS targets.
• Star-shaped bounding box features plus a refinement branch yield further AP gains by capturing box geometry and context.
• VFNet consistently surpasses FCOS+ATSS by about 2.0 AP across backbones, with VFNet-X-1200 achieving 55.1 AP on COCO test-dev.
• VFNet-X with strong backbones (e.g., Res2Net-101-DCN) achieves state-of-the-art single-model single-scale performance on COCO test-dev (55.1 AP).
• The Varifocal Loss is generally beneficial across detectors (ResNet-50, RetinaNet, FoveaBox, RepPoints, ATSS) and outperforms Generalized Focal Loss in experiments.