[Paper Review] How close are we to understanding image-based saliency?

Matthias Kümmerer, Thomas S. A. Wallis+1 more|arXiv (Cornell University)|Sep 26, 2014

Visual Attention and Saliency Detection39 references6 citations

TL;DR

This paper re-evaluates image-based saliency by modeling fixations as point processes to compute log-likelihoods, revealing that state-of-the-art models capture only about one-third of explainable spatial information. It introduces a principled method to identify where and why models fail, challenging the notion that spatial saliency is nearly understood.

ABSTRACT

Within the set of the many complex factors driving gaze placement, the properities of an image that are associated with fixations under free viewing conditions have been studied extensively. There is a general impression that the field is close to understanding this particular association. Here we frame saliency models probabilistically as point processes, allowing the calculation of log-likelihoods and bringing saliency evaluation into the domain of information. We compared the information gain of state-of-the-art models to a gold standard and find that only one third of the explainable spatial information is captured. We additionally provide a principled method to show where and how models fail to capture information in the fixations. Thus, contrary to previous assertions, purely spatial saliency remains a significant challenge.

Motivation & Objective

• To assess how well current saliency models explain human fixation patterns under free viewing conditions.
• To challenge the prevailing belief that spatial saliency is nearly fully understood by quantifying the remaining unexplained information.
• To develop a principled method for identifying spatial regions where models fail to predict fixations.
• To frame saliency evaluation in information-theoretic terms using log-likelihoods of point process models.

Proposed method

• Modeling human fixations as a point process to enable probabilistic evaluation using log-likelihoods.
• Using a gold-standard fixation dataset as a benchmark for information gain calculation.
• Comparing state-of-the-art saliency models against the gold standard using information-theoretic metrics.
• Applying spatial decomposition to identify regions of high model error and quantify information loss.
• Calculating explainable spatial information as the total information in the gold standard data.
• Using likelihood-based metrics to assess model performance beyond traditional correlation measures.

Experimental results

Research questions

• RQ1 How much of the explainable spatial information in human fixations is captured by current state-of-the-art saliency models?
• RQ2 To what extent do existing models fail to predict fixations in specific spatial regions?
• RQ3 Can a principled method be developed to localize and quantify model failures in saliency prediction?
• RQ4 Is the assumption that spatial saliency is nearly understood supported by information-theoretic evidence?

Key findings

• State-of-the-art saliency models capture only approximately one-third of the explainable spatial information in human fixation data.
• The information-theoretic evaluation reveals significant unexplained variance, indicating that purely spatial saliency remains a major challenge.
• Model failures are not uniformly distributed but are concentrated in specific spatial regions, which can be systematically identified.
• The use of point process modeling enables a more rigorous and principled evaluation of saliency models compared to traditional metrics.
• The gold standard fixation data contains significantly more information than current models can explain, highlighting a large performance gap.
• The method provides a diagnostic tool to analyze model shortcomings, offering a path toward improved saliency modeling.