[Paper Review] Deep Contrast Learning for Salient Object Detection

Guanbin Li, Yizhou Yu|arXiv (Cornell University)|Mar 7, 2016

Visual Attention and Saliency DetectionComputer Science42 references132 citations

TL;DR

Two-stream end-to-end deep network (pixel-level MS-FCN and segment-level pooling) learns visual contrast for salient object detection; optional fully connected CRF post-processing improves spatial coherence.

ABSTRACT

Salient object detection has recently witnessed substantial progress due to powerful features extracted using deep convolutional neural networks (CNNs). However, existing CNN-based methods operate at the patch level instead of the pixel level. Resulting saliency maps are typically blurry, especially near the boundary of salient objects. Furthermore, image patches are treated as independent samples even when they are overlapping, giving rise to significant redundancy in computation and storage. In this CVPR 2016 paper, we propose an end-to-end deep contrast network to overcome the aforementioned limitations. Our deep network consists of two complementary components, a pixel-level fully convolutional stream and a segment-wise spatial pooling stream. The first stream directly produces a saliency map with pixel-level accuracy from an input image. The second stream extracts segment-wise features very efficiently, and better models saliency discontinuities along object boundaries. Finally, a fully connected CRF model can be optionally incorporated to improve spatial coherence and contour localization in the fused result from these two streams. Experimental results demonstrate that our deep model significantly improves the state of the art.

Motivation & Objective

• Motivate robust salient object detection beyond patch-based CNNs by modeling visual contrast at pixel and segment levels.
• Propose an end-to-end architecture that produces high-resolution saliency maps efficiently.
• Allow boundary-aware refinement via a fully connected CRF on the fused outputs.

Proposed method

• Introduce a two-stream architecture: a pixel-level multi-scale fully convolutional network (MS-FCN) that produces a dense saliency map, and a segment-level spatial pooling stream that computes saliency over superpixels efficiently.
• Fuse the two saliency maps via a 1x1 convolution layer whose weights are learned.
• Optionally refine the fused map with a fully connected CRF to improve spatial coherence and contour localization.
• Train streams in alternation: initialize the segment stream, then jointly fine-tune both streams and the fusion layer with a cross-entropy loss against ground-truth saliency maps.
• Use an 8-pixel stride MS-FCN with hole (à trous) convolutions to maintain resolution and multi-scale context.
• Define a loss weighting beta_i to balance salient and non-salient pixel contributions in training.

Experimental results

Research questions

• RQ1 Can an end-to-end architecture combining pixel-level and segment-level cues outperform patch-based CNNs in salient object detection?
• RQ2 Does incorporating a CRF post-processing step yield measurable gains in spatial coherence and boundary accuracy?
• RQ3 How do multi-scale contextual features and segment-level masking contribute to saliency accuracy across diverse datasets?
• RQ4 Is fused MS-FCN and segment-level saliency more robust across images with multiple or boundary-touching salient objects?

Key findings

• The DCL (two-stream) model outperforms prior methods on multiple datasets in maxF, MAE, and precision-recall analyses.
• Adding CRF refinement (DCL +) yields further gains in accuracy and contour preservation across datasets.
• The MS-FCN stream contributes substantially to performance, with the full two-stream fusion providing the best results.
• The proposed method achieves state-of-the-art results versus eight recent methods and a FCN baseline across MSRA-B, HKU-IS, DUT-OMRON, PASCAL-S, and SOD datasets.
• Training is feasible (approximately 25 hours on MSRA-B) and testing is efficient (approximately 1.5 seconds per image for DCL; 0.8 seconds for CRF refinement).
• Ablation studies show both deep contrast learning and CRF contribute to improvements, with the two streams complementing each other.