[Paper Review] HSCS: Hierarchical Sparsity Based Co-saliency Detection for RGBD Images

Runmin Cong, Jianjun Lei+4 more|arXiv (Cornell University)|Nov 16, 2018

Visual Attention and Saliency Detection67 references4 citations

TL;DR

This paper proposes HSCS, a hierarchical sparsity-based co-saliency detection method for RGBD images that leverages global and pairwise sparsity reconstruction to model inter-image relationships, combined with energy function refinement for improved smoothness and consistency. The method achieves state-of-the-art performance on two RGBD co-saliency benchmarks, outperforming existing approaches both quantitatively and qualitatively, with a significant 8.4% F-measure gain when depth is incorporated.

ABSTRACT

Co-saliency detection aims to discover common and salient objects in an image group containing more than two relevant images. Moreover, depth information has been demonstrated to be effective for many computer vision tasks. In this paper, we propose a novel co-saliency detection method for RGBD images based on hierarchical sparsity reconstruction and energy function refinement. With the assistance of the intra saliency map, the inter-image correspondence is formulated as a hierarchical sparsity reconstruction framework. The global sparsity reconstruction model with a ranking scheme focuses on capturing the global characteristics among the whole image group through a common foreground dictionary. The pairwise sparsity reconstruction model aims to explore the corresponding relationship between pairwise images through a set of pairwise dictionaries. In order to improve the intra-image smoothness and inter-image consistency, an energy function refinement model is proposed, which includes the unary data term, spatial smooth term, and holistic consistency term. Experiments on two RGBD co-saliency detection benchmarks demonstrate that the proposed method outperforms the state-of-the-art algorithms both qualitatively and quantitatively.

Motivation & Objective

• To address the challenge of detecting common salient objects across multiple RGBD images, where depth information is underutilized in existing co-saliency methods.
• To improve inter-image correspondence modeling by capturing both global and local relationships among image groups.
• To enhance intra-image smoothness and inter-image consistency through an energy function refinement model.
• To develop an efficient co-saliency detection framework that reduces computational cost compared to matching- or propagation-based methods.

Proposed method

• A hierarchical sparsity reconstruction model is proposed, consisting of a global sparsity reconstruction with a ranking scheme to capture common foreground features across the entire image group.
• Pairwise sparsity reconstruction is employed to model local correspondence between image pairs using dedicated pairwise foreground dictionaries.
• A multi-component energy function is introduced, integrating unary data, spatial smoothness, and holistic consistency terms to refine saliency maps.
• Foreground seeds are selected using a ranking scheme that filters out background regions, improving the robustness of dictionary learning.
• Depth information is used both as a constraint in inter-image correspondence modeling and as a color feature supplement in the refinement stage.
• Superpixels are used to partition images, and the number of initial foreground seeds is optimized to balance reconstruction accuracy and background noise.

Experimental results

Research questions

• RQ1 How can depth information be effectively integrated into co-saliency detection for RGBD image groups to improve accuracy?
• RQ2 What is the impact of using a hierarchical sparsity model combining global and pairwise reconstruction on inter-image correspondence modeling?
• RQ3 How does the proposed energy function refinement model enhance intra-image smoothness and inter-image consistency?
• RQ4 To what extent does the ranking scheme for foreground seed selection improve robustness against background interference?
• RQ5 How does the proposed method compare in efficiency and accuracy to existing state-of-the-art co-saliency detection methods on RGBD datasets?

Key findings

• The inclusion of depth information leads to an 8.4% relative improvement in F-measure on the RGBD CoSal150 dataset, demonstrating its effectiveness in enhancing co-saliency detection.
• The ranking scheme for foreground seed selection significantly improves performance by filtering out background regions, as evidenced by qualitative results and quantitative gains.
• The optimal number of initial foreground seeds is 40 per image, as performance degrades beyond this point due to increased background noise.
• The method achieves a mean running time of 8.29 seconds per image, significantly faster than matching-based methods like MCLP (41.03 sec) and ICS (42.67 sec).
• The optimal number of superpixels is 400, though performance remains stable across a range of values, indicating low sensitivity to this parameter.
• The proposed HSCS method achieves state-of-the-art performance on both the RGBD CoSal150 and NYU-DepthV2 co-saliency detection benchmarks, outperforming existing methods in both qualitative and quantitative evaluations.