[Paper Review] Contrast-Oriented Deep Neural Networks for Salient Object Detection

Guanbin Li, Yizhou Yu|arXiv (Cornell University)|Mar 30, 2018

Visual Attention and Saliency Detection62 references3 citations

TL;DR

This paper proposes a hybrid contrast-oriented deep neural network for salient object detection, combining a multi-scale fully convolutional network (MS-FCN) for dense pixel-wise saliency prediction and a segment-level spatial pooling stream for regional contrast modeling. An attentional fusion module adaptively combines predictions from both streams, while a customized fully connected CRF with salient contour features enhances spatial coherence and boundary accuracy, achieving state-of-the-art performance on six benchmark datasets.

ABSTRACT

Deep convolutional neural networks have become a key element in the recent breakthrough of salient object detection. However, existing CNN-based methods are based on either patch-wise (region-wise) training and inference or fully convolutional networks. Methods in the former category are generally time-consuming due to severe storage and computational redundancies among overlapping patches. To overcome this deficiency, methods in the second category attempt to directly map a raw input image to a predicted dense saliency map in a single network forward pass. Though being very efficient, it is arduous for these methods to detect salient objects of different scales or salient regions with weak semantic information. In this paper, we develop hybrid contrast-oriented deep neural networks to overcome the aforementioned limitations. Each of our deep networks is composed of two complementary components, including a fully convolutional stream for dense prediction and a segment-level spatial pooling stream for sparse saliency inference. We further propose an attentional module that learns weight maps for fusing the two saliency predictions from these two streams. A tailored alternate scheme is designed to train these deep networks by fine-tuning pre-trained baseline models. Finally, a customized fully connected CRF model incorporating a salient contour feature embedding can be optionally applied as a post-processing step to improve spatial coherence and contour positioning in the fused result from these two streams. Extensive experiments on six benchmark datasets demonstrate that our proposed model can significantly outperform the state of the art in terms of all popular evaluation metrics.

Motivation & Objective

• To overcome the limitations of patch-wise CNNs, which suffer from high computational and storage overhead due to redundant processing of overlapping regions.
• To address the shortcomings of fully convolutional networks (FCNs), which struggle with multi-scale objects and weak semantic regions due to lack of regional context and poor boundary localization.
• To improve saliency detection by explicitly modeling visual contrast at both pixel and segment levels using complementary deep learning streams.
• To enhance spatial coherence and contour accuracy through a customized fully connected CRF with salient contour feature embedding.
• To achieve state-of-the-art performance across diverse benchmarks using an end-to-end, trainable framework with joint optimization of dual streams and post-processing.

Proposed method

• A multi-scale fully convolutional network (MS-FCN) is used to predict dense saliency maps by leveraging multi-scale receptive fields to capture visual contrast at different spatial scales.
• A segment-level spatial pooling stream processes superpixels from multiple image segmentation levels to compute sparse saliency maps based on contrast between adjacent regions.
• An attentional fusion module learns dynamic weights to combine predictions from the MS-FCN and segment-level streams, enabling adaptive integration of pixel-level and regional contrast information.
• A customized fully connected CRF is applied as a post-processing step, incorporating a salient contour feature embedding to refine spatial coherence and improve boundary localization.
• The model is trained using a tailored alternate fine-tuning scheme on pre-trained baseline models, enabling effective joint optimization of the dual-stream architecture.
• Multi-scale input feeding (scaling factors 1, 0.75, 0.5) is employed in the MS-FCN stream to improve robustness to objects of varying sizes.

Experimental results

Research questions

• RQ1 Can a hybrid deep network architecture combining dense and sparse prediction streams improve salient object detection beyond current fully convolutional or patch-based methods?
• RQ2 How effectively can multi-scale feature learning and segment-level contrast modeling jointly enhance detection of salient objects across diverse scales and with weak semantic content?
• RQ3 To what extent does an attentional fusion mechanism improve the integration of pixel-level and region-level saliency predictions?
• RQ4 Does incorporating salient contour features into a CRF post-processing step significantly improve boundary accuracy and spatial coherence?
• RQ5 Can the proposed framework achieve consistent state-of-the-art performance across multiple benchmark datasets under standard evaluation metrics?

Key findings

• The proposed model achieves a maximum F-measure improvement of 3.62% and a 7.32% reduction in MAE on the DUT-OMRON dataset when using ResNet-101 instead of VGG-16 in the MS-FCN stream.
• Adopting multi-scale inputs (scaling factors 1, 0.75, 0.5) improves the maximum F-measure by 2.46% and reduces MAE by 6.58% on DUT-OMRON.
• Using multi-level image segmentation (three segmentation levels) increases the maximum F-measure by 0.88% and reduces MAE by 1.40% on DUT-OMRON.
• Incorporating contour-guided CRF post-processing yields a 1.50% improvement in maximum F-measure and an 8.57% reduction in MAE compared to standard CRF on the VGG-16 based model.
• The ResNet-101 based model with multi-scale inputs and contour-guided CRF achieves a maximum F-measure of 89.6% and MAE of 0.048 on DUT-OMRON, outperforming prior state-of-the-art methods.
• Extensive experiments on six benchmark datasets confirm that the proposed method consistently outperforms existing methods across all standard evaluation metrics, including F-measure, MAE, and E-measure.