[Paper Review] Visual Saliency Based on Multiscale Deep Features

Guanbin Li, Yizhou Yu|arXiv (Cornell University)|Mar 30, 2015

Visual Attention and Saliency DetectionComputer Science33 references259 citations

TL;DR

This paper proposes a deep learning-based visual saliency model that leverages multiscale deep features from pre-trained CNNs to predict saliency maps with high accuracy. By combining multiscale features from nested image regions, fully connected layers for regression, spatial coherence refinement, and multi-level segmentation fusion, the method achieves state-of-the-art performance, improving F-Measure by 13.2% and reducing MAE by 35.1% on a new large-scale benchmark dataset (HKU-IS).

ABSTRACT

Visual saliency is a fundamental problem in both cognitive and computational sciences, including computer vision. In this CVPR 2015 paper, we discover that a high-quality visual saliency model can be trained with multiscale features extracted using a popular deep learning architecture, convolutional neural networks (CNNs), which have had many successes in visual recognition tasks. For learning such saliency models, we introduce a neural network architecture, which has fully connected layers on top of CNNs responsible for extracting features at three different scales. We then propose a refinement method to enhance the spatial coherence of our saliency results. Finally, aggregating multiple saliency maps computed for different levels of image segmentation can further boost the performance, yielding saliency maps better than those generated from a single segmentation. To promote further research and evaluation of visual saliency models, we also construct a new large database of 4447 challenging images and their pixelwise saliency annotation. Experimental results demonstrate that our proposed method is capable of achieving state-of-the-art performance on all public benchmarks, improving the F-Measure by 5.0% and 13.2% respectively on the MSRA-B dataset and our new dataset (HKU-IS), and lowering the mean absolute error by 5.7% and 35.1% respectively on these two datasets.

Motivation & Objective

• To develop a deep learning-based visual saliency model that effectively captures multiscale contextual contrasts using pre-trained CNN features.
• To address the limitations of existing saliency models by incorporating spatial coherence and multi-level image segmentation for improved localization and robustness.
• To construct a new, large-scale, and challenging benchmark dataset (HKU-IS) with 4,447 images and pixel-wise saliency annotations to support advanced saliency research.
• To demonstrate that multiscale deep features from pre-trained ImageNet networks are highly effective for saliency prediction, outperforming traditional handcrafted features.

Proposed method

• Extracts multiscale CNN features from three nested rectangular windows around each image region: the region itself, its immediate neighbors, and the full image.
• Uses a fully connected neural network to regress saliency scores from concatenated multiscale CNN features, trained on labeled saliency maps.
• Applies a spatial coherence refinement module to enhance the smoothness and structural consistency of predicted saliency maps.
• Fuses saliency maps generated from 15 levels of image segmentation to improve detection accuracy and robustness.
• Trains the end-to-end model using a regression loss on a large set of annotated saliency maps to optimize for F-Measure and mean absolute error (MAE).
• Employs a pre-trained ImageNet CNN as the feature extractor to leverage rich semantic information from deep hierarchical representations.

Experimental results

Research questions

• RQ1 Can multiscale deep features from pre-trained CNNs significantly improve visual saliency prediction compared to traditional methods?
• RQ2 How effective is the integration of spatial coherence and multi-level image segmentation in refining saliency predictions?
• RQ3 To what extent do multiscale features capture region contrast and semantic context for better saliency estimation?
• RQ4 How does the performance of deep learning-based saliency models scale on more complex, diverse, and challenging datasets?
• RQ5 Can a new large-scale benchmark dataset (HKU-IS) better support the evaluation and advancement of state-of-the-art saliency models?

Key findings

• The proposed method achieves a 13.2% improvement in F-Measure on the new HKU-IS dataset compared to the second-best method, significantly outperforming existing models.
• On the MSRA-B dataset, the method improves F-Measure by 5.0% and reduces MAE by 5.7% compared to the second-best baseline.
• The model reduces MAE by 35.1% on the challenging HKU-IS dataset, demonstrating superior performance on complex scenes with multiple salient objects.
• The use of spatial coherence refinement increases both precision and recall, with clear visual and quantitative improvements in saliency map smoothness and accuracy.
• Fusing saliency maps from 15 segmentation levels boosts average precision by 2.15% and recall by 3.47% compared to the best single-level segmentation.
• Ablation studies confirm that all three components of the multiscale feature vector (A, B, C) are complementary, with the full S-3CNN feature set yielding the best performance.