[Paper Review] Progressive Feature Polishing Network for Salient Object Detection

Bo Wang, Quan Chen+4 more|arXiv (Cornell University)|Nov 14, 2019

Visual Attention and Saliency Detection10 citations

TL;DR

This paper proposes the Progressive Feature Polishing Network (PFPN), a simple yet effective framework that iteratively refines multi-level features in a recurrent, parallel manner using Feature Polishing Modules (FPMs). By directly integrating high-level semantic information into all lower-level features at each stage, PFPN reduces long-term dependency issues and achieves state-of-the-art performance on five benchmark datasets without post-processing, with results improving monotonically with more polishing steps.

ABSTRACT

Feature matters for salient object detection. Existing methods mainly focus on designing a sophisticated structure to incorporate multi-level features and filter out cluttered features. We present Progressive Feature Polishing Network (PFPN), a simple yet effective framework to progressively polish the multi-level features to be more accurate and representative. By employing multiple Feature Polishing Modules (FPMs) in a recurrent manner, our approach is able to detect salient objects with fine details without any post-processing. A FPM parallelly updates the features of each level by directly incorporating all higher level context information. Moreover, it can keep the dimensions and hierarchical structures of the feature maps, which makes it flexible to be integrated with any CNN-based models. Empirical experiments show that our results are monotonically getting better with increasing number of FPMs. Without bells and whistles, PFPN outperforms the state-of-the-art methods significantly on five benchmark datasets under various evaluation metrics.

Motivation & Objective

• To address the limitations of existing salient object detection methods that suffer from information loss and long-term dependency due to indirect multi-level feature integration.
• To develop a simple, generalizable framework that enhances feature representation quality across all levels without complex architectural modifications.
• To eliminate the need for post-processing by achieving high-fidelity predictions directly through iterative feature refinement.
• To preserve hierarchical structure and dimensionality during feature refinement, enabling seamless integration with any CNN-based backbone.

Proposed method

• PFPN employs a recurrent architecture with multiple Feature Polishing Modules (FPMs) applied in sequence to progressively refine multi-level features.
• Each FPM updates all feature levels in parallel by directly fusing higher-level semantic features into lower-level features, preserving their original dimensions and hierarchy.
• The FPM design avoids cascaded or skip-connection-based feature aggregation, reducing long-term dependency problems common in U-Net or recursive refinement approaches.
• The framework uses transition modules (TMs) to align feature dimensions between the backbone and FPMs, ensuring compatibility across levels.
• Feature fusion in FPMs is performed via element-wise concatenation and convolution, maintaining simplicity while enabling effective information flow.
• The entire network is end-to-end trainable, with no need for post-processing like DenseCRF, as the model learns to produce high-quality saliency maps directly.

Experimental results

Research questions

• RQ1 Can a simple, recurrent feature refinement mechanism improve multi-level representation quality in salient object detection without complex architectures?
• RQ2 How does progressive polishing of features across multiple stages affect detection accuracy and detail recovery?
• RQ3 Does direct integration of high-level semantics into all lower-level features outperform indirect or sequential feature aggregation methods?
• RQ4 To what extent does the proposed method reduce the need for post-processing techniques like DenseCRF?
• RQ5 How does the number of polishing steps influence performance, and does performance converge with increasing iterations?

Key findings

• PFPN with 2 FPMs achieves state-of-the-art performance on five benchmark datasets, outperforming 16 SOTA methods across multiple metrics including MAE, max F, mean F, and S.
• The model's performance improves monotonically with an increasing number of FPMs, with the largest gain observed between T=0 and T=1, and marginal improvement beyond T=3.
• Even without any post-processing, PFPN achieves better results than other methods that use DenseCRF, indicating its superior intrinsic feature learning capability.
• The use of shared weights across FPMs leads to performance degradation compared to separate weights, showing that stage-specific refinement is more effective.
• Visualization confirms that saliency maps become progressively sharper and more accurate with each polishing step, with fine details and cluttered regions being systematically refined.