[Paper Review] Instance-Level Relative Saliency Ranking with Graph Reasoning

Nian Liu, Long Li+3 more|arXiv (Cornell University)|Jul 8, 2021

Visual Attention and Saliency Detection87 references4 citations

TL;DR

This paper proposes the first end-to-end model for instance-level relative saliency ranking, combining improved Mask R-CNN for instance segmentation with a graph reasoning module that integrates local contrast, global context, instance interactions, and semantic priors to infer relative saliency ranks. The method achieves state-of-the-art performance on a newly introduced dataset with a novel evaluation metric, demonstrating superior accuracy and practical utility in applications like adaptive image retargeting.

ABSTRACT

Conventional salient object detection models cannot differentiate the importance of different salient objects. Recently, two works have been proposed to detect saliency ranking by assigning different degrees of saliency to different objects. However, one of these models cannot differentiate object instances and the other focuses more on sequential attention shift order inference. In this paper, we investigate a practical problem setting that requires simultaneously segment salient instances and infer their relative saliency rank order. We present a novel unified model as the first end-to-end solution, where an improved Mask R-CNN is first used to segment salient instances and a saliency ranking branch is then added to infer the relative saliency. For relative saliency ranking, we build a new graph reasoning module by combining four graphs to incorporate the instance interaction relation, local contrast, global contrast, and a high-level semantic prior, respectively. A novel loss function is also proposed to effectively train the saliency ranking branch. Besides, a new dataset and an evaluation metric are proposed for this task, aiming at pushing forward this field of research. Finally, experimental results demonstrate that our proposed model is more effective than previous methods. We also show an example of its practical usage on adaptive image retargeting.

Motivation & Objective

• To address the gap in existing saliency detection models that treat all salient objects equally, despite real-world scenes having objects with varying degrees of saliency.
• To develop a unified, end-to-end framework that simultaneously segments salient instances and infers their relative saliency ranking.
• To model inter-instance relationships and saliency cues (local, global, semantic) effectively for accurate ranking.
• To propose a new dataset and evaluation metric tailored for instance-level relative saliency ranking to support future research.

Proposed method

• An improved Mask R-CNN is used as the backbone for instance segmentation, providing precise masks for salient objects.
• A novel graph reasoning module is introduced, combining four graphs: instance interaction, local contrast, global contrast, and high-level semantic prior.
• The graph reasoning module aggregates multi-level features through message passing to model complex relationships between salient instances.
• A new loss function is designed to supervise the saliency ranking branch, optimizing for relative rank order accuracy.
• The model is trained in a two-stage manner: first instance segmentation, then saliency ranking with joint optimization.
• Saliency ranking is predicted as a continuous score per instance, enabling relative comparison of object importance.

Experimental results

Research questions

• RQ1 Can an end-to-end deep learning model simultaneously perform accurate instance-level saliency segmentation and relative saliency ranking?
• RQ2 How can multi-level contextual cues—local contrast, global context, instance interaction, and semantic priors—be effectively integrated for relative saliency prediction?
• RQ3 What loss function design best supports learning accurate relative ranking without relying on absolute saliency supervision?
• RQ4 How does the proposed graph reasoning module improve ranking performance compared to sequential attention or pixel-wise regression?
• RQ5 Can the proposed model be effectively applied to real-world tasks such as adaptive image retargeting?

Key findings

• The proposed model achieves state-of-the-art performance on the newly introduced dataset, outperforming existing methods in both segmentation and ranking accuracy.
• The graph reasoning module significantly improves ranking performance by effectively modeling inter-instance relationships and multi-scale context.
• The proposed loss function enables better optimization for relative ranking, leading to more accurate saliency order prediction compared to standard regression loss.
• The model demonstrates practical utility in adaptive image retargeting, preserving the most salient objects under large-scale reductions while removing less important ones.
• Ablation studies confirm the importance of each component in the graph reasoning module, with the full model yielding the best results.
• The new dataset and evaluation metric are shown to be effective in measuring instance-level relative saliency ranking performance, supporting future benchmarking.