[Paper Review] Cascaded Partial Decoder for Fast and Accurate Salient Object Detection

Zhe Wu, Li Su+1 more|arXiv (Cornell University)|Apr 18, 2019

Visual Attention and Saliency Detection43 references100 citations

TL;DR

Introduces a cascaded partial decoder (CPD) that discards low-level high-resolution features to boost speed, while using a generated saliency map to refine deeper features for accuracy, achieving state-of-the-art results on five benchmarks and faster inference.

ABSTRACT

Existing state-of-the-art salient object detection networks rely on aggregating multi-level features of pre-trained convolutional neural networks (CNNs). Compared to high-level features, low-level features contribute less to performance but cost more computations because of their larger spatial resolutions. In this paper, we propose a novel Cascaded Partial Decoder (CPD) framework for fast and accurate salient object detection. On the one hand, the framework constructs partial decoder which discards larger resolution features of shallower layers for acceleration. On the other hand, we observe that integrating features of deeper layers obtain relatively precise saliency map. Therefore we directly utilize generated saliency map to refine the features of backbone network. This strategy efficiently suppresses distractors in the features and significantly improves their representation ability. Experiments conducted on five benchmark datasets exhibit that the proposed model not only achieves state-of-the-art performance but also runs much faster than existing models. Besides, the proposed framework is further applied to improve existing multi-level feature aggregation models and significantly improve their efficiency and accuracy.

Motivation & Objective

• Motivate reducing computation by discarding low-level high-resolution features in deep saliency models.
• Propose a two-branch bifurcated backbone with partial decoders for initial and refined saliency maps.
• Introduce a holistic attention module to expand saliency coverage and suppress distractors.
• Develop a cascaded optimization mechanism where the initial map guides refinement of deeper features.
• Demonstrate that CPD improves both accuracy and speed on multiple benchmarks and can enhance existing models.

Proposed method

• Adopts a bifurcated backbone where initial attention is produced from top-3 feature levels (f3, f4, f5).
• Implements a partial decoder that aggregates only deep-level features to generate an initial saliency map S_i.
• Generates a holistic attention map S_h to refine mid-level features by element-wise multiplication, yielding f3^d.
• Uses a second partial decoder on refined features to produce the final saliency map S_d.
• Introduces a fast context module inspired by receptive field blocks to capture global context with four branches and channel reduction.
• Jointly trains two branches with two saliency maps via a combined cross-entropy loss to promote consistent saliency discovery.

Experimental results

Research questions

• RQ1 Can discarding low-level features reduce computation without sacrificing saliency accuracy?
• RQ2 Does using an initial saliency map to refine higher-level features improve final saliency quality?
• RQ3 Does a holistic attention module improve object coverage and boundary delineation compared to initial attention?
• RQ4 Can the CPD framework improve existing deep aggregation models beyond its own architecture?
• RQ5 Are two-branch saliency designs viable for real-time salient object detection across benchmarks?

Key findings

• CPD achieves state-of-the-art performance on five benchmarks (ECSSD, HKU-IS, PASCAL-S, DUTS, DUT-OMRON).
• CPD runs faster than existing models, with notable speedups demonstrated in FPS comparisons (e.g., CPD vs. many baselines on DUTS).
• The attention branch (S_i) and the cascaded refinement (S_d) jointly outperform single-branch decoders across datasets.
• Embedding CPD into existing models (BMPM, Amulet, NLDF) yields substantial accuracy and efficiency gains (CPD-CPD-A variants show improvements).
• Holistic attention outperforms initial attention in multiple models, and choosing Conv3_3 as optimization layer provides a balance of accuracy and speed.