[Paper Review] Feature Pyramid Networks for Object Detection

Tsung-Yi Lin, Piotr Dollár+4 more|arXiv (Cornell University)|Dec 9, 2016

Advanced Neural Network Applications34 references454 citations

TL;DR

This paper introduces Feature Pyramid Networks (FPNs) that build a fast, semantically strong multi-scale feature pyramid inside a single ConvNet, improving detection and segmentation while keeping single-scale testing efficiency. It achieves state-of-the-art single-model results on COCO with Faster R-CNN and Faster R-CNN variants, using only a single input image scale.

ABSTRACT

Feature pyramids are a basic component in recognition systems for detecting objects at different scales. But recent deep learning object detectors have avoided pyramid representations, in part because they are compute and memory intensive. In this paper, we exploit the inherent multi-scale, pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost. A top-down architecture with lateral connections is developed for building high-level semantic feature maps at all scales. This architecture, called a Feature Pyramid Network (FPN), shows significant improvement as a generic feature extractor in several applications. Using FPN in a basic Faster R-CNN system, our method achieves state-of-the-art single-model results on the COCO detection benchmark without bells and whistles, surpassing all existing single-model entries including those from the COCO 2016 challenge winners. In addition, our method can run at 5 FPS on a GPU and thus is a practical and accurate solution to multi-scale object detection. Code will be made publicly available.

Motivation & Objective

• Motivate robust object detection across a wide range of object scales.
• Leverage the inherent pyramidal feature hierarchy of ConvNets to create high-level semantic features at all scales.
• Develop a fast, end-to-end trainable feature pyramid that avoids costly image pyramids.

Proposed method

• Introduce bottom-up, top-down, and lateral connections to create a multi-scale feature pyramid (P2–P5) from backbone feature maps (C2–C5).
• Upsample high-level semantic maps and fuse with corresponding lower-level maps via 1x1 lateral connections and 3x3 refinements.
• Attach identical detection heads (shared across pyramid levels) to each pyramid level for RPN and Fast R-CNN, enabling multi-scale proposals and detections.
• Assign RoIs to pyramid levels based on their spatial extent using a log-scale mapping similar to grid-based multi-scale detectors.
• Train end-to-end on a single image scale, avoiding the memory burden of featurized image pyramids.
• Extend the pyramid to segmentation proposals with small MLP heads on each pyramid level.

Experimental results

Research questions

• RQ1 Can a top-down, lateral-connection based pyramid built inside a ConvNet provide rich semantic features across multiple scales without the cost of image pyramids?
• RQ2 Do multi-scale predictions on all pyramid levels improve region proposals and object detection, particularly for small objects, compared to single-scale baselines?
• RQ3 Is sharing the detection heads across pyramid levels effective, and how does feature sharing affect training and inference time?

Key findings

• FPN significantly improves region proposal recall and detection metrics over single-scale baselines (e.g., AR 1k improves by 8.0 points; COCO-style AP improves by 2.3 points; PASCAL-style AP by 3.8 points against a strong single-scale Faster R-CNN baseline).
• Using a top-down enrichment with lateral connections yields higher-quality multi-scale features and outperforms variants without these connections.
• When integrated with RPN and Fast/Faster R-CNN, FPN achieves competitive or superior APs on COCO minival/test-dev, surpassing then-state-of-the-art single-model results without image pyramids.
• The method runs at modest speed, achieving around 6 FPS on a GPU for detection tasks with a lightweight, end-to-end trainable pyramid.
• FPN-based systems show significant gains for small objects (APs on small objects improve notably), while maintaining or improving overall speed compared to single-scale baselines.
• The framework generalizes to segmentation proposals, improving both proposal quality and speed over image-pyramid-based methods.