[Paper Review] Amulet: Aggregating Multi-level Convolutional Features for Salient Object Detection

Pingping Zhang, Dong Wang+3 more|arXiv (Cornell University)|Aug 7, 2017

Visual Attention and Saliency Detection45 references116 citations

TL;DR

Amulet proposes a multi-level feature aggregation framework (AmuletNet) that fuses multi-resolution FCN features with bidirectional recursive supervision and boundary-preserving refinements to advance salient object detection. It achieves state-of-the-art results across major datasets and runs near real-time.

ABSTRACT

Fully convolutional neural networks (FCNs) have shown outstanding performance in many dense labeling problems. One key pillar of these successes is mining relevant information from features in convolutional layers. However, how to better aggregate multi-level convolutional feature maps for salient object detection is underexplored. In this work, we present Amulet, a generic aggregating multi-level convolutional feature framework for salient object detection. Our framework first integrates multi-level feature maps into multiple resolutions, which simultaneously incorporate coarse semantics and fine details. Then it adaptively learns to combine these feature maps at each resolution and predict saliency maps with the combined features. Finally, the predicted results are efficiently fused to generate the final saliency map. In addition, to achieve accurate boundary inference and semantic enhancement, edge-aware feature maps in low-level layers and the predicted results of low resolution features are recursively embedded into the learning framework. By aggregating multi-level convolutional features in this efficient and flexible manner, the proposed saliency model provides accurate salient object labeling. Comprehensive experiments demonstrate that our method performs favorably against state-of-the art approaches in terms of near all compared evaluation metrics.

Motivation & Objective

• Motivate effective utilization of multi-level convolutional features for salient object detection beyond last-layer semantics.
• Develop a framework that aggregates features into multiple resolutions to balance coarse semantics and fine details.
• Introduce recursive supervision to enable bidirectional information flow and enhance boundary accuracy.
• Incorporate edge-aware low-level features and low-resolution predictions to refine object boundaries.
• Demonstrate strong generalization across datasets and efficient inference on GPUs.

Proposed method

• Build AmuletNet on a VGG-16 backbone with five conv blocks; remove the last pooling to obtain features at conv1-2, conv2-2, conv3-3, conv4-3, conv5-3.
• Introduce Resolution-Based Feature Integration (RFC) to resize and fuse multi-level features into multiple resolutions simultaneously.
• Use Deep Recursive Supervision (DRS) with autoregressive connections and deeply supervised losses to enable bidirectional information flow among predictions.
• Apply Boundary Preserved Refinement (BPR) by incorporating edge-aware low-level features to refine final saliency boundaries.
• Infer the final saliency map by fusing multiple level predictions and an adaptive mean-contrast fusion of foreground/background excitation maps.

Experimental results

Research questions

• RQ1 Can multi-level, multi-resolution feature aggregation improve saliency detection beyond single-level high-level features?
• RQ2 Does bidirectional recursive supervision enable better information sharing across levels and improve boundary accuracy?
• RQ3 Do edge-aware low-level features and boundary refinements meaningfully enhance saliency boundary localization?
• RQ4 Is AmuletNet generalizable across diverse saliency datasets despite training on MSRA10K?

Key findings

• AmuletNet achieves state-of-the-art F-measure and MAE across major datasets (DUTS-TE, ECSSD, HKU-IS, PASCAL-S, SOD) compared to64+ baselines.
• Boundary preserved refinements (BPR) significantly improve MAE and boundary accuracy, with ablations showing MAE drops without BPR.
• Multi-level feature integration with resolutions improves performance over single-resolution baselines; higher resolutions yield better results.
• Deep Recursive Supervision enables bidirectional information flow and outperforms traditional deeply supervised networks in saliency tasks.
• The model trained on MSRA10K generalizes well to other large-scale datasets, surpassing methods pretrained on target datasets.
• The approach runs at approximately 16 frames per second on modern GPUs.