[Paper Review] Weakly Supervised Top-down Salient Object Detection.

Hisham Cholakkal, Jubin Johnson+1 more|arXiv (Cornell University)|Nov 16, 2016

Visual Attention and Saliency Detection53 references3 citations

TL;DR

This paper proposes a weakly supervised top-down saliency framework that uses only binary image-level labels to generate saliency maps, leveraging backtracked CNN confidence contributions and combining them with selected bottom-up saliency maps. It achieves state-of-the-art performance on seven datasets across four applications, outperforming fully supervised methods despite minimal supervision.

ABSTRACT

Top-down saliency models produce a probability map that peaks at target locations specified by a task/goal such as object detection. They are usually trained in a fully supervised setting involving pixel-level annotations of objects. We propose a weakly supervised top-down saliency framework using only binary labels that indicate the presence/absence of an object in an image. First, the probabilistic contribution of each image region to the confidence of a CNN-based image classifier is computed through a backtracking strategy to produce top-down saliency. From a set of saliency maps of an image produced by fast bottom-up saliency approaches, we select the best saliency map suitable for the top-down task. The selected bottom-up saliency map is combined with the top-down saliency map. Features having high combined saliency are used to train a linear SVM classifier to estimate feature saliency. This is integrated with combined saliency and further refined through a multi-scale superpixel-averaging of saliency map. We evaluate the performance of the proposed weakly supervised top-down saliency against fully supervised approaches and achieve state-of-the-art performance. Experiments are carried out on seven challenging datasets and quantitative results are compared with 36 closely related approaches across 4 different applications.

Motivation & Objective

• To develop a top-down saliency detection framework that requires only binary image-level labels instead of costly pixel-level annotations.
• To enable top-down saliency learning in a weakly supervised setting by leveraging the probabilistic contribution of image regions to classifier confidence.
• To improve saliency map quality by selecting the most suitable bottom-up saliency map for each image and combining it with top-down saliency.
• To refine saliency predictions through multi-scale superpixel averaging and train a linear SVM on high-saliency features for improved localization.
• To achieve state-of-the-art performance on multiple benchmark datasets with minimal supervision, demonstrating the effectiveness of the weakly supervised approach.

Proposed method

• Compute top-down saliency by backtracking the contribution of each image region to the confidence score of a CNN-based image classifier.
• Select the best-performing bottom-up saliency map from multiple fast bottom-up approaches based on relevance to the top-down task.
• Combine the selected bottom-up saliency map with the top-down saliency map to produce a fused saliency map.
• Train a linear SVM on features with high combined saliency to estimate feature-level saliency and improve localization accuracy.
• Refine the final saliency map through multi-scale superpixel averaging to enhance spatial consistency and precision.
• Use the combined saliency and refined features to train a classifier that predicts object presence with weak supervision.

Experimental results

Research questions

• RQ1 Can top-down saliency detection be effectively learned using only binary image-level labels instead of pixel-level annotations?
• RQ2 How can the contribution of image regions to classifier confidence be leveraged to generate meaningful top-down saliency maps in a weakly supervised setting?
• RQ3 What is the optimal way to combine top-down saliency with bottom-up saliency to improve localization accuracy?
• RQ4 Can superpixel-based averaging improve the quality and robustness of weakly supervised saliency maps?
• RQ5 Does the proposed framework achieve state-of-the-art performance compared to fully supervised methods on diverse benchmark datasets?

Key findings

• The proposed weakly supervised top-down saliency framework achieves state-of-the-art performance on seven challenging benchmark datasets.
• The method outperforms 36 closely related approaches across four different applications, demonstrating strong generalization under weak supervision.
• The integration of top-down saliency from backtracked CNN confidence with selected bottom-up saliency maps significantly improves localization accuracy.
• Multi-scale superpixel averaging of the saliency map enhances spatial consistency and refines object boundary localization.
• The linear SVM trained on high-combined-saliency features effectively learns discriminative patterns with minimal supervision.
• The framework achieves competitive performance compared to fully supervised methods despite using only binary image-level labels.