[Paper Review] Salient Objects in Clutter: Bringing Salient Object Detection to the Foreground

Deng-Ping Fan, Ming‐Ming Cheng+4 more|arXiv (Cornell University)|Mar 16, 2018

Visual Attention and Saliency Detection39 references43 citations

TL;DR

The paper introduces the SOC dataset (Salient Objects in Clutter) to address data bias in salient object detection, provides a comprehensive benchmark of CNN-based SOD models on this dataset, and analyzes performance across multiple real-world attributes.

ABSTRACT

We provide a comprehensive evaluation of salient object detection (SOD) models. Our analysis identifies a serious design bias of existing SOD datasets which assumes that each image contains at least one clearly outstanding salient object in low clutter. The design bias has led to a saturated high performance for state-of-the-art SOD models when evaluated on existing datasets. The models, however, still perform far from being satisfactory when applied to real-world daily scenes. Based on our analyses, we first identify 7 crucial aspects that a comprehensive and balanced dataset should fulfill. Then, we propose a new high quality dataset and update the previous saliency benchmark. Specifically, our SOC (Salient Objects in Clutter) dataset, includes images with salient and non-salient objects from daily object categories. Beyond object category annotations, each salient image is accompanied by attributes that reflect common challenges in real-world scenes. Finally, we report attribute-based performance assessment on our dataset.

Motivation & Objective

• Identify biases in existing SOD datasets that overestimate performance in idealized, low-clutter scenes.
• Create a realistic, large-scale SOD dataset (SOC) including salient and non-salient images with instance-level annotations and attributes.
• Benchmark major CNN-based SOD models on SOC to reveal generalization gaps and guide future research.
• Provide attribute-based performance analysis to understand model strengths/weaknesses under real-world challenges.

Proposed method

• Define seven criteria for a realistic and balanced SOD dataset.
• Assemble SOC with 6,000 images (3,000 salient, 3,000 non-salient) across 80+ categories, including non-salient images and instance-level annotations.
• Annotate salient objects with high-quality pixel-level masks and provide per-image attributes (e.g., motion blur, occlusion, clutter).
• Evaluate representative single-task and multi-task CNN-based SOD models on SOC using pixel-wise accuracy, region similarity (F-measure), and structure similarity (S-measure).
• Conduct attribute-based performance evaluation to analyze model performance under specific scene challenges.
• Release the dataset and benchmarking tools publicly.

Experimental results

Research questions

• RQ1 How do current SOD models perform on realistic scenes with clutter and non-salient images, compared to existing benchmarks?
• RQ2 What dataset design biases exist in prior SOD datasets, and how does SOC address them?
• RQ3 How do salient-object attributes (e.g., motion blur, occlusion, clutter) affect model performance across different architectures?
• RQ4 Can attribute-based benchmarking reveal weaknesses of state-of-the-art SOD models and guide future research directions?

Key findings

• SOC is the largest instance-level SOD dataset at the time of publication, with 6,000 images (3,000 salient, 3,000 non-salient) across 80+ categories.
• SOC includes high-quality instance-level saliency masks and object attributes reflecting real-world challenges, enabling richer analysis than prior datasets.
• Benchmark results show that top-performing models on existing datasets do not achieve satisfactory performance on SOC, highlighting a realism gap.
• Attribute-based evaluation demonstrates how performance degrades for challenges like large objects, clutter, occlusion, and other specified attributes, guiding future model improvements.
• Multi-task and weakly supervised models show promise but still lag behind fully supervised single-task models on SOC, suggesting directions for robust, real-world SOD.