[Paper Review] Parallel Residual Bi-Fusion Feature Pyramid Network for Accurate Single-Shot Object Detection

Ping-Yang Chen, Ming‐Ching Chang+2 more|arXiv (Cornell University)|Dec 3, 2020

Advanced Neural Network Applications69 references118 citations

TL;DR

This paper proposes Parallel Residual Bi-Fusion Feature Pyramid Network (PRB-FPN), a novel one-stage object detection framework that enhances feature fusion through parallel, bi-directional (top-down and bottom-up) pathways using a novel Concatenation and Re-organization (CORE) module and residual design. PRB-FPN achieves state-of-the-art accuracy on MS COCO and UAVDT17, with 48.9% AP on COCO using CSPDarknet-53 and 59.4% AP on 800×800 input, outperforming YOLOv7 and other SoTA models while maintaining real-time inference capability.

ABSTRACT

This paper proposes the Parallel Residual Bi-Fusion Feature Pyramid Network (PRB-FPN) for fast and accurate single-shot object detection. Feature Pyramid (FP) is widely used in recent visual detection, however the top-down pathway of FP cannot preserve accurate localization due to pooling shifting. The advantage of FP is weakened as deeper backbones with more layers are used. In addition, it cannot keep up accurate detection of both small and large objects at the same time. To address these issues, we propose a new parallel FP structure with bi-directional (top-down and bottom-up) fusion and associated improvements to retain high-quality features for accurate localization. We provide the following design improvements: (1) A parallel bifusion FP structure with a bottom-up fusion module (BFM) to detect both small and large objects at once with high accuracy. (2) A concatenation and re-organization (CORE) module provides a bottom-up pathway for feature fusion, which leads to the bi-directional fusion FP that can recover lost information from lower-layer feature maps. (3) The CORE feature is further purified to retain richer contextual information. Such CORE purification in both top-down and bottom-up pathways can be finished in only a few iterations. (4) The adding of a residual design to CORE leads to a new Re-CORE module that enables easy training and integration with a wide range of deeper or lighter backbones. The proposed network achieves state-of-the-art performance on the UAVDT17 and MS COCO datasets. Code is available at https://github.com/pingyang1117/PRBNet_PyTorch.

Motivation & Objective

• To address the limitation of standard Feature Pyramid Networks (FPNs) in accurately detecting both small and large objects due to resolution loss from pooling.
• To improve feature representation by introducing a bi-directional fusion mechanism that preserves both high-level semantic and low-level spatial details.
• To enhance training stability and model integration by incorporating a residual design into the fusion pathway.
• To achieve high detection accuracy with real-time inference speed, suitable for deployment on edge devices.

Proposed method

• Proposes a parallel bi-fusion structure that simultaneously fuses three feature pyramid levels (P3, P4, P5) using a bottom-up fusion module (BFM) to improve detection of both small and large objects.
• Introduces the Concatenation and Re-organization (CORE) module to recursively fuse features from adjacent layers, enabling efficient bottom-up information flow and contextual feature recovery.
• Employs a residual design in the CORE module (Re-CORE) to stabilize training and allow seamless integration with various backbones, including deeper and lighter architectures.
• Uses a parallel architecture to avoid the memory explosion of traditional concatenation-based fusion, maintaining efficiency while enhancing feature quality.
• Applies the CORE module in both top-down and bottom-up pathways to purify features iteratively, preserving rich contextual and localization information.
• Employs a multi-path fusion strategy that enables the network to generate three prediction maps in parallel, improving detection accuracy without sacrificing speed.

Experimental results

Research questions

• RQ1 Can a parallel, bi-directional feature fusion mechanism improve detection accuracy for both small and large objects in single-shot detectors?
• RQ2 How does the CORE module enhance feature representation by enabling recursive bottom-up fusion and contextual information recovery?
• RQ3 To what extent does the residual design in the CORE module improve training stability and backbone compatibility?
• RQ4 Does the proposed PRB-FPN achieve state-of-the-art performance on standard benchmarks like MS COCO and UAVDT17 while maintaining real-time inference speed?
• RQ5 How does PRB-FPN compare to existing bi-directional FPNs like BiFPN and PANet in terms of accuracy and efficiency?

Key findings

• PRB-FPN achieves 48.9% AP on MS COCO test-dev using CSPDarknet-53 with 800×800 input, surpassing YOLOv7-E6E and other SoTA one-stage detectors.
• On the same benchmark, PRB-FPN achieves 46.1% AP with ResNet-50 and 800×800 input, outperforming previous SoTA models like AB+FSAF and Cascade R-CNN.
• The PRB-FPN6 variant achieves 55.9% AP on COCO for large objects (APL), indicating strong performance on challenging, large-scale instances.
• The model maintains real-time inference speed, achieving 15.9 FPS with ResNet-50 and 11.6 FPS with CSPDarknet-53 on 800×800 input, demonstrating efficiency for edge deployment.
• Visual comparisons show PRB-FPN6 outperforms YOLOv7-E6E in foggy and occluded scenes, successfully detecting small and heavily obscured objects.
• Extensive ablation studies confirm that the CORE module and residual design significantly improve feature quality and training convergence.