[Paper Review] Deeply-Supervised Recurrent Convolutional Neural Network for Saliency Detection

Youbao Tang, Xiangqian Wu+1 more|arXiv (Cornell University)|Aug 18, 2016

Visual Attention and Saliency Detection35 references4 citations

TL;DR

This paper proposes a deeply-supervised recurrent convolutional neural network (DSRCNN) for saliency detection, integrating recurrent connections in each convolutional layer to enhance contextual feature learning and using side-output layers for deep supervision to improve feature discrimination. The model achieves state-of-the-art performance on five benchmark datasets by fusing multi-scale predictions to generate high-quality saliency maps.

ABSTRACT

This paper proposes a novel saliency detection method by developing a deeply-supervised recurrent convolutional neural network (DSRCNN), which performs a full image-to-image saliency prediction. For saliency detection, the local, global, and contextual information of salient objects is important to obtain a high quality salient map. To achieve this goal, the DSRCNN is designed based on VGGNet-16. Firstly, the recurrent connections are incorporated into each convolutional layer, which can make the model more powerful for learning the contextual information. Secondly, side-output layers are added to conduct the deeply-supervised operation, which can make the model learn more discriminative and robust features by effecting the intermediate layers. Finally, all of the side-outputs are fused to integrate the local and global information to get the final saliency detection results. Therefore, the DSRCNN combines the advantages of recurrent convolutional neural networks and deeply-supervised nets. The DSRCNN model is tested on five benchmark datasets, and experimental results demonstrate that the proposed method significantly outperforms the state-of-the-art saliency detection approaches on all test datasets.

Motivation & Objective

• To improve saliency detection by effectively capturing local, global, and contextual features in images.
• To address the limitations of standard CNNs in modeling long-range dependencies and contextual information.
• To enhance feature learning through deep supervision using intermediate side-outputs.
• To fuse multi-level predictions for robust and accurate saliency map generation.
• To achieve superior performance on benchmark datasets compared to existing state-of-the-art methods.

Proposed method

• The DSRCNN is built upon VGGNet-16 as the backbone architecture.
• Recurrent connections are introduced into each convolutional layer to progressively refine feature representations and model long-range dependencies.
• Side-output layers are added at multiple intermediate stages to enable deep supervision and improve training stability.
• Each side-output produces a saliency map, and all are fused to generate the final prediction.
• The network is trained with a multi-task loss combining supervision from all side-outputs and the final output.
• Feature learning is enhanced through end-to-end optimization with residual connections and skip connections from the VGG-16 base.

Experimental results

Research questions

• RQ1 Can recurrent connections in convolutional layers improve contextual feature learning for saliency detection?
• RQ2 Does deep supervision via intermediate side-outputs lead to more robust and discriminative features?
• RQ3 Can fusing multi-scale predictions from different network depths improve saliency map quality?
• RQ4 How does the proposed DSRCNN compare to state-of-the-art methods on standard benchmarks?
• RQ5 To what extent does the integration of local, global, and contextual features enhance detection accuracy?

Key findings

• The DSRCNN significantly outperforms existing state-of-the-art saliency detection methods on all five benchmark datasets.
• The use of recurrent connections improves feature representation by modeling long-range dependencies across feature maps.
• Deep supervision via side-outputs enhances feature discrimination and network robustness during training.
• Fusing predictions from multiple stages leads to more accurate and detailed saliency maps.
• The model achieves superior performance in terms of mean absolute error (MAE), F-measure, and precision-recall metrics across all datasets.
• The ablation study confirms the effectiveness of both recurrent connections and deep supervision in boosting performance.