[Paper Review] AugFPN: Improving Multi-scale Feature Learning for Object Detection

Chaoxu Guo, Bin Fan+3 more|arXiv (Cornell University)|Dec 11, 2019

Advanced Neural Network ApplicationsComputer Science52 references42 citations

TL;DR

AugFPN introduces Consistent Supervision, Residual Feature Augmentation, and Soft RoI Selection to address defects in FPN, yielding consistent AP gains across backbones and detectors on COCO.

ABSTRACT

Current state-of-the-art detectors typically exploit feature pyramid to detect objects at different scales. Among them, FPN is one of the representative works that build a feature pyramid by multi-scale features summation. However, the design defects behind prevent the multi-scale features from being fully exploited. In this paper, we begin by first analyzing the design defects of feature pyramid in FPN, and then introduce a new feature pyramid architecture named AugFPN to address these problems. Specifically, AugFPN consists of three components: Consistent Supervision, Residual Feature Augmentation, and Soft RoI Selection. AugFPN narrows the semantic gaps between features of different scales before feature fusion through Consistent Supervision. In feature fusion, ratio-invariant context information is extracted by Residual Feature Augmentation to reduce the information loss of feature map at the highest pyramid level. Finally, Soft RoI Selection is employed to learn a better RoI feature adaptively after feature fusion. By replacing FPN with AugFPN in Faster R-CNN, our models achieve 2.3 and 1.6 points higher Average Precision (AP) when using ResNet50 and MobileNet-v2 as backbone respectively. Furthermore, AugFPN improves RetinaNet by 1.6 points AP and FCOS by 0.9 points AP when using ResNet50 as backbone. Codes will be made available.

Motivation & Objective

• Identify design defects in the FPN feature pyramid that hinder multi-scale feature exploitation.
• Propose AugFPN with three components to address semantic gap, information loss, and RoI assignment ineffectiveness.
• Evaluate AugFPN across multiple detectors and backbones on MS COCO to assess robustness and generality.
• Demonstrate significant AP improvements over FPN baselines and show compatibility with one- and two-stage detectors.

Proposed method

• Consistent Supervision to enforce similar semantic information across multi-scale feature maps before fusion.
• Residual Feature Augmentation to incorporate ratio-invariant contextual features into the highest-level pyramid map (M5) via a residual branch and Adaptive Spatial Fusion.
• Soft RoI Selection to learn adaptive RoI feature fusion from all pyramid levels using Adaptive Spatial Fusion, avoiding heuristic level assignment.

Experimental results

Research questions

• RQ1 Can Consistent Supervision reduce semantic gaps between pyramid levels prior to fusion?
• RQ2 Does enriching the highest-level feature with ratio-invariant context reduce information loss and improve multi-scale fusion?
• RQ3 Can adaptive, learnable RoI feature fusion across all pyramid levels outperform heuristic RoI level assignment and max/sum fusion?
• RQ4 Do the proposed components generalize across backbones and detectors on COCO?

Key findings

• Replacing FPN with AugFPN improves Faster R-CNN with ResNet50 by 2.3 AP (to 38.8 AP).
• AugFPN also improves Faster R-CNN with ResNet101 by 1.7 AP (to 40.6 AP) and with ResNext-101 variants by up to 1.4 AP.
• MobileNet-V2 backbone with AugFPN yields a 1.6 AP gain for Faster R-CNN.
• One-stage detectors also benefit: RetinaNet gains 1.6 AP (ResNet50 or 1.3 AP with MobileNet-v2), and FCOS gains 0.9 AP with ResNet-50.
• Mask R-CNN sees detection gains of 2.0 AP (ResNet50) and 1.5 AP (ResNet101) when using AugFPN, with corresponding segmentation gains.