[Paper Review] Concatenated Feature Pyramid Network for Instance Segmentation

Yongqing Sun, Pranav Shenoy K P+2 more|arXiv (Cornell University)|Mar 16, 2019

Advanced Neural Network Applications20 references4 citations

TL;DR

This paper proposes a Concatenated Feature Pyramid Network (CFPN) that enhances instance segmentation by fusing low-level features (edges, textures) with high-level semantic features across all pyramid levels using concatenation and a novel inception-like module. The method improves mask quality and AP by 0.9 over Mask R-CNN on COCO, with minimal computational overhead and no additional parameters beyond the backbone.

ABSTRACT

Low level features like edges and textures play an important role in accurately localizing instances in neural networks. In this paper, we propose an architecture which improves feature pyramid networks commonly used instance segmentation networks by incorporating low level features in all layers of the pyramid in an optimal and efficient way. Specifically, we introduce a new layer which learns new correlations from feature maps of multiple feature pyramid levels holistically and enhances the semantic information of the feature pyramid to improve accuracy. Our architecture is simple to implement in instance segmentation or object detection frameworks to boost accuracy. Using this method in Mask RCNN, our model achieves consistent improvement in precision on COCO Dataset with the computational overhead compared to the original feature pyramid network.

Motivation & Objective

• To improve instance segmentation accuracy by optimally integrating low-level features (edges, textures) into high-level semantic features across all pyramid levels.
• To overcome the limitations of element-wise addition in existing feature pyramid networks, which restricts feature correlation learning and fails to capture cross-level dependencies.
• To design a computationally efficient architecture that boosts mask quality without increasing model parameters or inference cost.
• To demonstrate that concatenation-based fusion outperforms element-wise addition in feature pyramid networks for instance segmentation.
• To validate the method on the COCO dataset using Mask R-CNN as the baseline framework.

Proposed method

• Introduces a new bottom-up pathway that propagates low-level features from deeper network layers to higher pyramid levels via concatenation, not element-wise addition.
• Employs a novel inception-like module that learns correlations between multi-level feature maps by concatenating features from different pyramid levels and applying convolutions.
• Uses 3x3 post-hoc convolution layers on concatenated feature maps to refine features and reduce aliasing, especially improving small object detection.
• Maintains the original top-down pathway with element-wise addition for stability and efficiency, while replacing lateral connections with concatenation in the bottom-up path.
• Applies the CFPN module to Mask R-CNN, replacing its standard FPN with a dual-pathway feature fusion mechanism: top-down and bottom-up with concatenation.
• Trains the model using standard optimization protocols on COCO with ResNet-50 backbone, batch normalization, and transfer learning from ImageNet.

Experimental results

Research questions

• RQ1 Can concatenation of multi-level features in a feature pyramid improve instance segmentation performance compared to element-wise addition?
• RQ2 Does incorporating low-level features across all pyramid levels enhance mask quality, especially for small objects?
• RQ3 Can a bottom-up feature fusion pathway with concatenation achieve better performance than existing top-down or hybrid approaches with minimal computational cost?
• RQ4 How does the proposed CFPN architecture compare to Mask R-CNN and PANet in terms of AP, AP50, and AP75 on the COCO dataset?
• RQ5 Does the addition of a post-hoc convolution layer significantly improve small object detection without increasing model complexity?

Key findings

• The proposed CFPN model achieves a mask AP of 34.8 on COCO, which is 0.9 higher than the baseline Mask R-CNN (33.9) and 0.4 higher than Mask R-CNN with bottom-up path augmentation (34.4).
• The model improves bounding box AP by 0.5 over Mask R-CNN and 0.3 over the BPA-enhanced version, demonstrating consistent performance gains.
• Model 1 achieves the highest performance in the AP_L (large objects) metric, indicating improved localization accuracy for larger instances due to better edge and texture utilization.
• The model without post-hoc convolution (Model 1*) achieves 34.6 mask AP with only 9.6×10⁹ MAC operations, showing that the core CFPN mechanism provides gains even without additional refinement layers.
• Despite using concatenation, which is typically more computationally expensive, the overall MAC count remains low (56.7×10⁹ for Model 1), only slightly above the original FPN (52.2×10⁹), proving computational efficiency.
• Visual comparisons show that masks generated by CFPN more accurately align with object boundaries, especially for small and complex objects, due to optimal low-level feature integration.