[Paper Review] Richer and Deeper Supervision Network for Salient Object Detection

Sen Jia, Neil D. B. Bruce|arXiv (Cornell University)|Jan 8, 2019

Visual Attention and Saliency Detection43 references17 citations

TL;DR

This paper proposes the Richer and Deeper Supervision (RDS) network for salient object detection, which enhances feature fusion across multi-scale outputs using richer global and local representations. By pre-training on object detection datasets (ImageNet-DET and PASCAL-VOC) to inject coarse global context, RDS achieves state-of-the-art performance on five benchmark datasets, with F-measure scores up to 0.953 and MAE as low as 0.028.

ABSTRACT

Recent Salient Object Detection (SOD) systems are mostly based on Convolutional Neural Networks (CNNs). Specifically, Deeply Supervised Saliency (DSS) system has shown it is very useful to add short connections to the network and supervising on the side output. In this work, we propose a new SOD system which aims at designing a more efficient and effective way to pass back global information. Richer and Deeper Supervision (RDS) is applied to better combine features from each side output without demanding much extra computational space. Meanwhile, the backbone network used for SOD is normally pre-trained on the object classification dataset, ImageNet. But the pre-trained model has been trained on cropped images in order to only focus on distinguishing features within the region of the object. But the ignored background information is also significant in the task of SOD. We try to solve this problem by introducing the training data designed for object detection. A coarse global information is learned based on an entire image with its bounding box before training on the SOD dataset. The large-scale of object images can slightly improve the performance of SOD. Our experiment shows the proposed RDS network achieves the state-of-the-art results on five public SOD datasets.

Motivation & Objective

• To address the limitation of shallow global feature propagation in existing deeply supervised networks for salient object detection.
• To improve feature fusion efficiency between multi-scale side outputs without significantly increasing computational cost.
• To mitigate the data scarcity issue in saliency detection by leveraging large-scale object detection datasets for pre-training.
• To enhance performance by incorporating background-aware features from object detection annotations, which are more informative than ImageNet classification data.
• To investigate the impact of pre-training on object detection data versus ImageNet classification data for saliency detection.

Proposed method

• Proposes a novel RDS network architecture that enhances feature fusion across side outputs using richer, deeper supervision paths to improve global and local feature integration.
• Introduces a two-stage training strategy: first fine-tuning on a merged object detection dataset (ImageNet-DET + PASCAL-VOC) to learn coarse location context, then fine-tuning on SOD datasets.
• Uses residual networks (ResNet-152) as the backbone, with side outputs at multiple layers to enable deep supervision and multi-level feature aggregation.
• Leverages bounding box annotations from object detection datasets to inject contextual and background-aware features during pre-training, improving generalization.
• Applies cross-entropy and L1 loss for supervision on side outputs, with a focus on preserving both high-level semantic and low-level edge details.
• Employs a two-stage fine-tuning process to mitigate catastrophic forgetting, using a low initial learning rate (0.001) on the object detection data before SOD fine-tuning.

Experimental results

Research questions

• RQ1 Can richer and deeper supervision mechanisms improve feature fusion in saliency detection networks without increasing computational cost?
• RQ2 Does pre-training on object detection datasets with full-image context improve saliency detection performance compared to ImageNet classification pre-training?
• RQ3 How does the inclusion of background and object context from bounding box annotations affect the detection of salient objects?
• RQ4 To what extent does catastrophic forgetting impact the transfer of knowledge from object detection to saliency detection when using multi-stage pre-training?
• RQ5 Can the proposed RDS network outperform existing state-of-the-art methods across diverse public SOD benchmarks?

Key findings

• The RDS-152 model achieved the highest F-measure of 0.953 on the HKU-IS dataset, outperforming all other state-of-the-art methods.
• On the DUT-OMRON dataset, RDS-152 achieved the best F-measure of 0.837 and lowest MAE of 0.050, ranking first among all compared methods.
• The RDS-152-OBJ variant, pre-trained on object detection data, achieved an F-measure of 0.818 on the PASCAL-SOD subset, surpassing other methods including AMU and SRM.
• The PR-curve analysis confirmed that RDS-152-OBJ consistently outperformed competitors across all five public SOD datasets, especially in high-recall regions.
• The ablation study showed that pre-training on object detection data led to a slight but consistent performance gain, particularly in boundary accuracy and global context understanding.
• Despite the slight improvement from object detection pre-training, the model still suffered from catastrophic forgetting, indicating a need for better continual learning strategies in future work.