[Paper Review] Saliency Prediction in the Deep Learning Era: An Empirical Investigation.

Ali Borji|arXiv (Cornell University)|Oct 8, 2018

Visual Attention and Saliency DetectionComputer Science46 citations

TL;DR

This paper conducts a comprehensive empirical investigation of deep learning-based visual saliency models, evaluating their performance across multiple image and video benchmarks. It identifies persistent gaps between model predictions and human attention, analyzes failure modes, and outlines key challenges and directions for next-generation saliency models.

ABSTRACT

Visual saliency models have enjoyed a big leap in performance in recent years, thanks to advances in deep learning and large scale annotated data. Despite enormous effort and huge breakthroughs, however, models still fall short in reaching human-level accuracy. In this work, I explore the landscape of the field emphasizing on new deep saliency models, benchmarks, and datasets. A large number of image and video saliency models are reviewed and compared over two image benchmarks and two large scale video datasets. Further, I identify factors that contribute to the gap between models and humans and discuss remaining issues that need to be addressed to build the next generation of more powerful saliency models. Some specific questions that are addressed include: in what ways current models fail, how to remedy them, what can be learned from cognitive studies of attention, how explicit saliency judgments relate to fixations, how to conduct fair model comparison, and what are the emerging applications of saliency models.

Motivation & Objective

• To evaluate the current state of deep learning-based visual saliency models using standardized benchmarks.
• To identify systematic failures of existing models in predicting human visual attention.
• To explore the relationship between explicit saliency judgments and eye fixation data.
• To establish fair comparison protocols for saliency models across image and video datasets.
• To highlight open challenges and emerging applications in saliency modeling.

Proposed method

• The study reviews and compares a large number of state-of-the-art deep saliency models on two image benchmarks and two large-scale video datasets.
• It employs standardized evaluation metrics to ensure fair and reproducible model comparisons.
• The analysis draws on cognitive science insights to interpret discrepancies between model predictions and human fixation patterns.
• The work evaluates both image and video saliency models using diverse, large-scale annotated datasets.
• It investigates the alignment between explicit saliency annotations and eye-tracking fixation data to assess model validity.
• The methodology includes systematic failure analysis to identify recurring weaknesses in model generalization and robustness.

Experimental results

Research questions

• RQ1 In what ways do current deep saliency models fail to predict human fixation patterns accurately?
• RQ2 How do explicit saliency judgments relate to actual human eye movements and fixations?
• RQ3 What factors contribute to the performance gap between deep learning models and human observers?
• RQ4 How can model comparisons be made fair and meaningful across diverse datasets and evaluation protocols?
• RQ5 What insights from cognitive science of attention can inform the design of more human-aligned saliency models?

Key findings

• Despite significant progress, deep saliency models still fall short of human-level accuracy in predicting visual attention.
• Models exhibit systematic failures in handling complex scenes, occlusions, and dynamic content, particularly in video settings.
• Explicit saliency judgments and fixation data show notable discrepancies, suggesting that not all saliency annotations are equivalent in capturing attention.
• Fair model comparison is challenging due to inconsistent evaluation protocols and dataset biases.
• Cognitive science insights reveal that attention mechanisms in humans involve top-down and bottom-up integration, which current models often fail to emulate effectively.
• Emerging applications of saliency models span visual analytics, robotics, and human-computer interaction, indicating growing practical relevance.