[Paper Review] Learning Spatial Fusion for Single-Shot Object Detection

Songtao Liu, Di Huang+1 more|arXiv (Cornell University)|Nov 21, 2019

Advanced Neural Network Applications44 references464 citations

TL;DR

The paper proposes adaptively spatial feature fusion (ASFF) to learn per-location fusion weights across feature pyramid levels, reducing cross-scale inconsistency in single-shot detectors and improving speed-accuracy trade-offs on COCO.

ABSTRACT

Pyramidal feature representation is the common practice to address the challenge of scale variation in object detection. However, the inconsistency across different feature scales is a primary limitation for the single-shot detectors based on feature pyramid. In this work, we propose a novel and data driven strategy for pyramidal feature fusion, referred to as adaptively spatial feature fusion (ASFF). It learns the way to spatially filter conflictive information to suppress the inconsistency, thus improving the scale-invariance of features, and introduces nearly free inference overhead. With the ASFF strategy and a solid baseline of YOLOv3, we achieve the best speed-accuracy trade-off on the MS COCO dataset, reporting 38.1% AP at 60 FPS, 42.4% AP at 45 FPS and 43.9% AP at 29 FPS. The code is available at https://github.com/ruinmessi/ASFF

Motivation & Objective

• Address scale variation in object detection using pyramidal feature representations.
• Mitigate inconsistency across feature pyramid levels during fusion in single-shot detectors.
• Provide a data-driven fusion mechanism that preserves efficiency and improves accuracy.

Proposed method

• Introduce ASFF to learn spatial fusion weights at each location across feature pyramid levels.
• Resize features from all levels to a common resolution before adaptive fusion.
• Compute fusion weights via softmax over learned parameters lambda, producing alpha, beta, gamma with alpha+beta+gamma=1.
• Fuse features per level as y^l_ij = alpha^l_ij x^{1->l}_ij + beta^l_ij x^{2->l}_ij + gamma^l_ij x^{3->l}_ij.
• Train end-to-end with standard detection losses plus existing YOLOv3 (and IoU) losses; keep inference overhead near-constant.
• Demonstrate compatibility with YOLOv3 and RetinaNet backbones and show improved speed-accuracy.

Experimental results

Research questions

• RQ1 Can adaptive, location-wise fusion across feature pyramid levels reduce the scale inconsistency in single-shot detectors?
• RQ2 How does ASFF compare to standard fusion (sum/concat) and ignore-region strategies in terms of accuracy and speed?
• RQ3 Does ASFF improve detection of small and medium objects without sacrificing large-object performance?
• RQ4 Is ASFF agnostic to backbone and detector design, and what is the practical inference overhead?

Key findings

• ASFF improves YOLOv3-based detector AP from 38.8% to 40.6% on COCO val-2017 (with minimal overhead).
• ASFF adds about 2 ms of inference time, maintaining around 46 FPS.
• ASFF yields better gains on small and medium objects (AP_S and AP_M increases of about 2.9 points each).
• RetinaNet with ASFF consistently improves AP over baselines (e.g., R50-FPN: 35.9 to 37.4; R101-FPN: 39.1 to 40.1 on val-2017).
• On COCO test-dev, YOLOv3+ASFF* reaches 42.4 AP at 63 FPS (and 43.9 AP at 29 FPS with ASFF* variant).