[Paper Review] Salient Object Detection in the Deep Learning Era: An In-Depth Survey

Wenguan Wang, Qiuxia Lai+4 more|arXiv (Cornell University)|Apr 19, 2019

Visual Attention and Saliency Detection86 citations

TL;DR

This paper provides a comprehensive survey of deep learning-based salient object detection (SOD), covering taxonomies, datasets, evaluation metrics, robustness to perturbations and adversarial attacks, cross-dataset generalization, and future directions. It also introduces a rich, annotated dataset and benchmarks to analyze SOD models.

ABSTRACT

As an essential problem in computer vision, salient object detection (SOD) has attracted an increasing amount of research attention over the years. Recent advances in SOD are predominantly led by deep learning-based solutions (named deep SOD). To enable in-depth understanding of deep SOD, in this paper, we provide a comprehensive survey covering various aspects, ranging from algorithm taxonomy to unsolved issues. In particular, we first review deep SOD algorithms from different perspectives, including network architecture, level of supervision, learning paradigm, and object-/instance-level detection. Following that, we summarize and analyze existing SOD datasets and evaluation metrics. Then, we benchmark a large group of representative SOD models, and provide detailed analyses of the comparison results. Moreover, we study the performance of SOD algorithms under different attribute settings, which has not been thoroughly explored previously, by constructing a novel SOD dataset with rich attribute annotations covering various salient object types, challenging factors, and scene categories. We further analyze, for the first time in the field, the robustness of SOD models to random input perturbations and adversarial attacks. We also look into the generalization and difficulty of existing SOD datasets. Finally, we discuss several open issues of SOD and outline future research directions.

Motivation & Objective

• Summarize how deep SOD models are categorized by network architecture, supervision level, learning paradigm, and object/instance focus.
• Analyze standard SOD datasets and evaluation metrics and their strengths/weaknesses.
• Provide an attribute-based evaluation using a novel annotated dataset to reveal model strengths and weaknesses.
• Assess robustness of SOD models to input perturbations and adversarial attacks.
• Discuss cross-dataset generalization, dataset difficulty, open issues, and future research directions.

Proposed method

• Proposes taxonomies for deep SOD models including network architectures (MLP, FCN, hybrid, capsule) and supervision levels (fully-, weakly-/unsupervised).
• Reviews learning paradigms (single-task vs. multi-task) and obj.-level vs. instance-level SOD.
• Constructs a rich attribute-annotated dataset to enable attribute-based performance evaluation and cross-model analysis.
• Examines robustness to random input perturbations and adversarial attacks to assess model reliability.
• Performs cross-dataset generalization studies and discusses dataset difficulty and generalization issues; provides open issues and future directions.

Experimental results

Research questions

• RQ1 What are the main architectural and supervision-based categories used to organize deep SOD models?
• RQ2 How do current SOD datasets and metrics compare, and what are their limitations?
• RQ3 How do SOD models perform across different attributes, scenes, and salient object types?
• RQ4 How robust are deep SOD models to input perturbations and adversarial attacks, and how well do models generalize across datasets?
• RQ5 What are the key open issues and promising directions for future SOD research?

Key findings

• The paper provides a systematic taxonomy of deep SOD models by network architecture, supervision level, learning paradigm, and object/instance level.
• An attribute-based evaluation is conducted using a novel dataset with rich annotations for object categories, scene categories, and challenging factors.
• The study analyzes robustness of SOD models to random input perturbations and conducts the first adversarial attack analysis for SOD models.
• Cross-dataset generalization analysis reveals dataset bias and varying difficulty across SOD benchmarks.
• The authors benchmark representative models and discuss open issues and future directions to guide ongoing research.
• All saliency maps, the annotated dataset, and evaluation code are made publicly available.