[Paper Review] AutoAssign: Differentiable Label Assignment for Dense Object Detection

Benjin Zhu, Jianfeng Wang+5 more|arXiv (Cornell University)|Jul 7, 2020

Advanced Neural Network Applications29 references194 citations

TL;DR

AutoAssign introduces a fully differentiable, appearance-aware label assignment for dense object detection, jointly optimizing spatial and scale assignment with two weighting modules (Center Weighting and Confidence Weighting) to outperform prior sampling strategies on MS COCO and other datasets.

ABSTRACT

Determining positive/negative samples for object detection is known as label assignment. Here we present an anchor-free detector named AutoAssign. It requires little human knowledge and achieves appearance-aware through a fully differentiable weighting mechanism. During training, to both satisfy the prior distribution of data and adapt to category characteristics, we present Center Weighting to adjust the category-specific prior distributions. To adapt to object appearances, Confidence Weighting is proposed to adjust the specific assign strategy of each instance. The two weighting modules are then combined to generate positive and negative weights to adjust each location's confidence. Extensive experiments on the MS COCO show that our method steadily surpasses other best sampling strategies by large margins with various backbones. Moreover, our best model achieves 52.1% AP, outperforming all existing one-stage detectors. Besides, experiments on other datasets, e.g., PASCAL VOC, Objects365, and WiderFace, demonstrate the broad applicability of AutoAssign.

Motivation & Objective

• Reduce reliance on hand-crafted label assignment by making positive/negative sampling fully data-driven.
• Develop category-specific priors for spatial center weighting to capture diverse object distributions.
• Develop instance-aware confidence weighting to adapt to appearance and scale of each object.
• Combine the two weighting modules to generate differentiable positive/negative weight maps for training.
• Demonstrate state-of-the-art performance on MS COCO and robustness across datasets.

Proposed method

• Adopt an anchor-free detector framework and treat all locations inside a ground-truth box across FPN levels as candidate positives and negatives during training.
• Center Weighting: learn category-specific, Gaussian-like priors G(d|μ,σ) to adapt the center prior per category and normalize by the downscaled distance to the box center.
• Confidence Weighting: compute a joint confidence P_i that combines classification confidence (with an Implicit-Objectness branch) and localization confidence; convert localization loss to a likelihood for seamless integration.
• Compute positive weights w_i^+ by normalizing the product C(P_i^+) (confidence) and G(d_i) across all candidate locations within the object’s region.
• Compute negative weights w_i^- using 1 - f(iou_i) where iou_i is the max IoU of location i with any GT box, ensuring a background emphasis for non-object regions.
• Define the final loss as L = -sum_n log(sum_{i in S_n} w_i^+ P_i^+) - sum_k log(1 - w_k^- P_k^-), with focal loss applied to negatives and a differentiable weighting mechanism during training that is not used at inference.
• Key equations include the Gaussian center weighting G(d|μ,σ) and the joint confidence formulation P_i^+, P_i^- with the normalization and weighting schemes (as detailed in the methodology).

Experimental results

Research questions

• RQ1 Can fully data-driven, differentiable label assignment improve detection performance over fixed or partially dynamic sampling strategies?
• RQ2 How do category-specific priors (Center Weighting) and instance-specific appearance/adaptation (Confidence Weighting) interact to optimize spatial and scale assignment?
• RQ3 Does the proposed weighting framework generalize across datasets with varying distributions (VOC, Objects365, WiderFace) without hyper-parameter tuning?
• RQ4 What is the impact of the Implicit-Objectness branch on filtering noise candidates and improving separation from background?
• RQ5 What is the upper-bound performance achievable by AutoAssign on standard benchmarks (e.g., MS COCO) with common backbones?

Key findings

• AutoAssign achieves state-of-the-art performance among one-stage detectors on MS COCO, with a representative best model achieving 52.1 AP on COCO test-dev under 2x schedules and multi-scale testing.
• Center Weighting substantially improves AP by aligning center priors with category-specific distributions (including cases where priors shift or elongate to match object shapes).
• Confidence Weighting further boosts AP by adaptively weighting spatial locations based on object appearance and scale, beyond center priors alone.
• The combination of Center Weighting and Confidence Weighting yields the best performance (AutoAssign), outperforming prior methods that rely on anchors, IoU thresholds, or top-k sampling.
• AutoAssign demonstrates robust generalization across datasets (PASCAL VOC, Objects365, WiderFace) without hyper-parameter re-tuning, outperforming fixed/partially dynamic methods on these distributions.