[Paper Review] Multi-patch Feature Pyramid Network for Weakly Supervised Object Detection in Optical Remote Sensing Images

Pourya Shamsolmoali, Jocelyn Chanussot+3 more|arXiv (Cornell University)|Aug 18, 2021

Remote-Sensing Image ClassificationEngineering58 references44 citations

TL;DR

This paper proposes MPFP-Net, a multi-patch feature pyramid network for weakly supervised object detection in optical remote sensing images. It introduces multiple patch learning to handle incomplete object instances and non-convex loss landscapes, while using norm-preserving, cross-scale feature fusion to enhance multi-scale detection. The model achieves state-of-the-art mAP of 86.73% on the LEVIR dataset with 51.2M parameters and 228B FLOPs, outperforming existing methods in accuracy and efficiency.

ABSTRACT

Object detection is a challenging task in remote sensing because objects only occupy a few pixels in the images, and the models are required to simultaneously learn object locations and detection. Even though the established approaches well perform for the objects of regular sizes, they achieve weak performance when analyzing small ones or getting stuck in the local minima (e.g. false object parts). Two possible issues stand in their way. First, the existing methods struggle to perform stably on the detection of small objects because of the complicated background. Second, most of the standard methods used hand-crafted features, and do not work well on the detection of objects parts of which are missing. We here address the above issues and propose a new architecture with a multiple patch feature pyramid network (MPFP-Net). Different from the current models that during training only pursue the most discriminative patches, in MPFPNet the patches are divided into class-affiliated subsets, in which the patches are related and based on the primary loss function, a sequence of smooth loss functions are determined for the subsets to improve the model for collecting small object parts. To enhance the feature representation for patch selection, we introduce an effective method to regularize the residual values and make the fusion transition layers strictly norm-preserving. The network contains bottom-up and crosswise connections to fuse the features of different scales to achieve better accuracy, compared to several state-of-the-art object detection models. Also, the developed architecture is more efficient than the baselines.

Motivation & Objective

• Address the challenge of detecting small or partially visible objects in optical remote sensing images due to limited supervision and complex backgrounds.
• Overcome the non-convexity of loss functions in weakly supervised learning that leads to false object part detection and incomplete instance learning.
• Improve feature representation for multi-scale object detection by fusing features across different levels using norm-preserving, cross-scale connections.
• Develop a scalable, efficient architecture that maintains high accuracy while reducing model size and FLOPs compared to state-of-the-art models.
• Enable robust detection across diverse image sources and conditions through joint patch-wise and scale-wise learning strategies.

Proposed method

• Proposes a multiple patch learning (MPL) scheme that divides image patches into class-affiliated subsets, enabling the model to learn co-occurring, discriminative parts of objects even without full-instance annotations.
• Introduces a novel feature fusion mechanism using bottom-up and crosswise connections to combine multi-scale features, with strictly norm-preserving fusion layers to stabilize gradient flow.
• Employs a weighted, instant fusion strategy that replaces computationally heavy softmax fusion, reducing training cost by 1.24× while maintaining performance.
• Incorporates a multi-directional, scale-wise feature pyramid network (ESS-FPN) that jointly scales image size, depth, and width to enhance feature representation across all dimensions.
• Uses a joint loss function to train the entire network end-to-end, integrating patch selection, feature aggregation, and semantic projection into a unified framework.
• Adopts SPN (Semantic Pyramid Network) as the backbone to extract richer multi-level features, improving both accuracy and inference speed.

Experimental results

Research questions

• RQ1 Can multiple patch learning improve weakly supervised object detection by modeling co-occurring object parts when full-instance annotations are unavailable?
• RQ2 How does cross-scale, norm-preserving feature fusion enhance detection performance for small and multi-scale objects in remote sensing images?
• RQ3 To what extent does joint scaling of image size, depth, and width improve feature representation and detection accuracy compared to single-dimension scaling?
• RQ4 Can the proposed instant fusion mechanism reduce computational cost without degrading model performance compared to softmax or dynamic fusion?
• RQ5 How does MPFP-Net compare in accuracy and efficiency to state-of-the-art models on benchmark remote sensing datasets?

Key findings

• MPFP-Net achieves a mean average precision (mAP) of 86.73% on the LEVIR test set, outperforming the baseline MPFP-Net-S6+FPN by 8.32 mAP and reducing parameters by 58%.
• The model is 3.7× faster than the next fastest baseline on a P40 GPU, with inference latency reduced by up to 3.8× compared to other detectors.
• The proposed instant fusion method runs 1.24× faster than softmax fusion while maintaining similar mAP and training stability, reducing computational cost without sacrificing performance.
• Joint multi-patch and multi-scale learning improves detection efficiency and accuracy, with MPFP-Net outperforming models using only patch learning or single-dimension scaling.
• On the DOTA dataset, MPFP-Net achieves state-of-the-art performance across 15 object classes, with precise and stable bounding box predictions even in cluttered backgrounds.
• Qualitative results on large-scale RSIs show that MPFP-Net generalizes well across different image sources and conditions, with robust detection of small and incomplete objects.