[Paper Review] Pyramid Feature Attention Network for Saliency detection

Ting Zhao, Xiangqian Wu|arXiv (Cornell University)|Mar 1, 2019

Visual Attention and Saliency Detection36 references60 citations

TL;DR

Pyramid Feature Attention Network (PFA) introduces context-aware pyramid feature extraction, channel-wise and spatial attention, plus an edge preservation loss to improve saliency detection boundaries, achieving state-of-the-art results on five benchmarks.

ABSTRACT

Saliency detection is one of the basic challenges in computer vision. How to extract effective features is a critical point for saliency detection. Recent methods mainly adopt integrating multi-scale convolutional features indiscriminately. However, not all features are useful for saliency detection and some even cause interferences. To solve this problem, we propose Pyramid Feature Attention network to focus on effective high-level context features and low-level spatial structural features. First, we design Context-aware Pyramid Feature Extraction (CPFE) module for multi-scale high-level feature maps to capture rich context features. Second, we adopt channel-wise attention (CA) after CPFE feature maps and spatial attention (SA) after low-level feature maps, then fuse outputs of CA & SA together. Finally, we propose an edge preservation loss to guide network to learn more detailed information in boundary localization. Extensive evaluations on five benchmark datasets demonstrate that the proposed method outperforms the state-of-the-art approaches under different evaluation metrics.

Motivation & Objective

• Motivate robust saliency detection by selecting effective high-level context features and preserving low-level spatial structure.
• Propose CPFE to capture multi-scale context for high-level features.
• Integrate channel-wise attention after CPFE and spatial attention on low-level features to refine saliency maps.
• Introduce an edge preservation loss to improve boundary localization.
• Demonstrate state-of-the-art performance across multiple datasets.

Proposed method

• Context-aware pyramid feature extraction (CPFE) uses atrous convolutions with dilation rates 3, 5, 7 on high-level VGG blocks to produce multi-scale context features, followed by cross-channel concatenation.
• Channel-wise attention (CA) weighting of high-level CPFE features to emphasize channels relevant to saliency.
• Spatial attention (SA) applied to low-level features to suppress background noise and sharpen boundaries.
• Fusion of CA-modulated high-level features and SA-modulated low-level features to generate the final saliency map.
• Edge preservation loss L_B guides boundary localization by supervising Laplacian-derived boundary maps with cross-entropy, combined with standard saliency loss L_S into L = α L_S + (1−α) L_B, with α explored to 0.7 as optimal.

Experimental results

Research questions

• RQ1 Can context-aware multi-scale high-level features combined with selective attention improve saliency detection compared to indiscriminate multi-scale fusion?
• RQ2 Do high-level channel-wise attention and low-level spatial attention improve saliency map quality and boundary accuracy?
• RQ3 Does an edge preservation loss enhance detail and boundary localization in saliency maps?
• RQ4 What is the optimal balance between saliency map accuracy and boundary detail within the loss function?

Key findings

• The proposed method achieves state-of-the-art performance on five benchmark datasets across metrics like wFβ and MAE.
• CPFE captures multi-scale context for high-level features, which, when coupled with channel-wise attention, improves salient region localization.
• Low-level features refined with spatial attention help preserve boundaries and suppress background noise.
• Edge preservation loss significantly improves boundary details, with α = 0.7 yielding the best MAE and wFβ on DUTS-test.
• Ablation studies show all components (CPFE, CA, SA, and edge preservation loss) contribute to performance gains; the full model yields the best results.
• Compared to prior methods, the approach shows strong robustness and particularly large gains on challenging datasets like DUT-OMRON.