[Paper Review] ssFPN: Scale Sequence (S^2) Feature Based-Feature Pyramid Network for Object Detection

Hye-Jin Park, Youngju Choi+2 more|arXiv (Cornell University)|Aug 24, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes ssFPN, a scale sequence (S²) feature pyramid network that enhances small object detection by applying 3D convolution across FPN's level axis to extract scale-invariant features. By treating FPN levels as a temporal sequence, it captures inter-level correlations and boosts high-resolution feature maps, improving AP by up to 2.0% on small objects in Faster R-CNN and 1.3% in YOLOv4-P5.

ABSTRACT

Feature Pyramid Network (FPN) has been an essential module for object detection models to consider various scales of an object. However, average precision (AP) on small objects is relatively lower than AP on medium and large objects. The reason is why the deeper layer of CNN causes information loss as feature extraction level. We propose a new scale sequence (S^2) feature extraction of FPN to strengthen feature information of small objects. We consider FPN structure as scale-space and extract scale sequence (S^2) feature by 3D convolution on the level axis of FPN. It is basically scale invariant feature and is built on high-resolution pyramid feature map for small objects. Furthermore, the proposed S^2 feature can be extended to most object detection models based on FPN. We demonstrate the proposed S2 feature can improve the performance of both one-stage and two-stage detectors on MS COCO dataset. Based on the proposed S2 feature, we achieve upto 1.3% and 1.1% of AP improvement for YOLOv4-P5 and YOLOv4-P6, respectively. For Faster RCNN and Mask R-CNN, we observe upto 2.0% and 1.6% of AP improvement with the suggested S^2 feature, respectively.

Motivation & Objective

• To address the persistent challenge of low average precision (AP) on small objects in object detection models.
• To overcome feature degradation and information loss in deeper CNN layers that disproportionately affect small object features.
• To improve feature representation across all object scales by leveraging inter-level correlations in the FPN feature pyramid.
• To design a plug-and-play module compatible with both one-stage and two-stage detectors based on FPN.

Proposed method

• Treat the FPN level axis as a time-like dimension, transforming 2D feature maps into a 3D tensor across levels for sequence modeling.
• Apply 3D convolution to extract scale sequence (S²) features that are inherently scale-invariant and capture multi-level context.
• Resize all FPN feature maps to a common high-resolution for consistent 3D convolution input, enhancing small object representation.
• Concatenate the extracted S² feature to the high-resolution feature map (e.g., P3) before detection head inference.
• Use 1x1 convolutions to adjust channel dimensions when integrating S² features into two-stage detectors.
• Design the S² module to be easily extendable to most FPN-based object detectors without architectural overhaul.

Experimental results

Research questions

• RQ1 Can modeling the FPN level axis as a sequence improve feature representation for small objects?
• RQ2 Does capturing inter-level correlations across FPN levels lead to better detection performance across all scales?
• RQ3 Can a 3D convolution-based S² feature extraction mechanism enhance scale invariance in object detection?
• RQ4 How does the S² feature perform across diverse detector architectures, including one-stage and two-stage models?
• RQ5 What is the inference efficiency and parameter overhead of integrating the S² feature into existing FPN-based detectors?

Key findings

• The proposed S² feature improved AP on small objects by up to 1.3% in YOLOv4-P5 and 1.1% in YOLOv4-P6.
• For Faster R-CNN and Mask R-CNN, the S² feature achieved a 2.0% and 1.6% AP improvement, respectively, with the highest gains on small objects.
• YOLOv4-P6 with S² feature saw the largest AP improvement on small objects (1.1%), followed by medium (0.3%) and large (0.4%) objects.
• The ablation study confirmed that concatenating the S² feature to P3 yielded the best performance with minimal computational overhead.
• Runtime analysis showed only a minor speed drop (1.7ms to 8.9ms) with batch size 8 on V100, indicating low inference cost.
• The S² feature improved AP50 and AP75 across all models, confirming consistent gains in detection robustness.