[Paper Review] F3Net: Fusion, Feedback and Focus for Salient Object Detection

Jun Wei, Shuhui Wang+1 more|arXiv (Cornell University)|Nov 26, 2019

Visual Attention and Saliency Detection35 references113 citations

TL;DR

F3 Net introduces a cross feature module for selective multi-level feature fusion, a cascaded feedback decoder for iterative refinement, and a pixel position aware loss to emphasize hard pixels, achieving state-of-the-art salient object detection across five datasets.

ABSTRACT

Most of existing salient object detection models have achieved great progress by aggregating multi-level features extracted from convolutional neural networks. However, because of the different receptive fields of different convolutional layers, there exists big differences between features generated by these layers. Common feature fusion strategies (addition or concatenation) ignore these differences and may cause suboptimal solutions. In this paper, we propose the F3Net to solve above problem, which mainly consists of cross feature module (CFM) and cascaded feedback decoder (CFD) trained by minimizing a new pixel position aware loss (PPA). Specifically, CFM aims to selectively aggregate multi-level features. Different from addition and concatenation, CFM adaptively selects complementary components from input features before fusion, which can effectively avoid introducing too much redundant information that may destroy the original features. Besides, CFD adopts a multi-stage feedback mechanism, where features closed to supervision will be introduced to the output of previous layers to supplement them and eliminate the differences between features. These refined features will go through multiple similar iterations before generating the final saliency maps. Furthermore, different from binary cross entropy, the proposed PPA loss doesn't treat pixels equally, which can synthesize the local structure information of a pixel to guide the network to focus more on local details. Hard pixels from boundaries or error-prone parts will be given more attention to emphasize their importance. F3Net is able to segment salient object regions accurately and provide clear local details. Comprehensive experiments on five benchmark datasets demonstrate that F3Net outperforms state-of-the-art approaches on six evaluation metrics.

Motivation & Objective

• Mitigate the discrepancy between multi-level CNN features in saliency detection (different receptive fields) to improve fusion quality.
• Introduce a selective fusion mechanism that suppresses redundant information while preserving complementary details.
• Refine multi-level features through iterative, cascaded feedback to sharpen boundaries and improve local details.
• Incorporate a loss that weights pixels by their local structure context to emphasize hard, boundary, and detail-rich regions.
• Demonstrate state-of-the-art performance on multiple benchmark datasets with comprehensive ablations.

Proposed method

• Propose Cross Feature Module (CFM) that fuses low- and high-level features via element-wise multiplication to extract common parts and then adds back original features to refine representations.
• Develop Cascaded Feedback Decoder (CFD) with multiple sub-decoders that perform bottom-up aggregation through CFM and top-down feature feedback to previous layers, forming a grid-like refinement network.
• Introduce Pixel Position Aware (PPA) loss which combines a location-aware weighted BCE with a weighted IoU loss to emphasize hard pixels and preserve local structure.
• Train with multi-level supervision (MLS) across CFD stages to stabilize optimization and enhance learning.
• Use ResNet-50 as backbone and a progressively refined decoding path to generate final saliency maps without post-processing.
• Evaluate on five datasets and perform ablations to show the contribution of CFM, CFD, and PPA.

Experimental results

Research questions

• RQ1 How can multi-level CNN features be fused more effectively to reduce redundancy and preserve semantic and spatial details in saliency maps?
• RQ2 Can a feedback-based decoding scheme iteratively refine features to recover boundary precision and local structures?
• RQ3 Does a pixel-position-aware loss improve detection of hard pixels and boundary regions compared to standard BCE or IoU losses?
• RQ4 What is the empirical impact of integrating MLS, CFM, and CFD on standard saliency metrics across benchmarks?

Key findings

• F3 Net achieves superior performance on five datasets across six metrics compared to state-of-the-art methods.
• CFM effectively suppresses background noises and sharpens high-level boundaries through selective fusion.
• CFD enables iterative refinement by feeding back refined features to previous layers, improving saliency map quality.
• PPA loss emphasizes hard pixels and incorporates local structure, improving boundary and detail representation.
• Ablation studies show that CFM, CFD, MLS, and PPA collectively contribute to the best results.
• The model delivers clearer saliency maps with reduced background noise and robust performance without post-processing.