[Paper Review] WEPSAM: Weakly Pre-Learnt Saliency Model

Avisek Lahiri, Sourya Roy+3 more|arXiv (Cornell University)|May 3, 2016

Visual Attention and Saliency Detection27 references4 citations

TL;DR

WEPSAM introduces a weakly pre-trained convolutional neural network for visual saliency prediction by leveraging ImageNet for domain-specific pre-training before fine-tuning on limited eye fixation data. The method achieves faster convergence and superior performance on MIT300 compared to state-of-the-art models, demonstrating that weak pre-training significantly improves saliency detection efficiency and accuracy.

ABSTRACT

Visual saliency detection tries to mimic human vision psychology which concentrates on sparse, important areas in natural image. Saliency prediction research has been traditionally based on low level features such as contrast, edge, etc. Recent thrust in saliency prediction research is to learn high level semantics using ground truth eye fixation datasets. In this paper we present, WEPSAM : Weakly Pre-Learnt Saliency Model as a pioneering effort of using domain specific pre-learing on ImageNet for saliency prediction using a light weight CNN architecture. The paper proposes a two step hierarchical learning, in which the first step is to develop a framework for weakly pre-training on a large scale dataset such as ImageNet which is void of human eye fixation maps. The second step refines the pre-trained model on a limited set of ground truth fixations. Analysis of loss on iSUN and SALICON datasets reveal that pre-trained network converges much faster compared to randomly initialized network. WEPSAM also outperforms some recent state-of-the-art saliency prediction models on the challenging MIT300 dataset.

Motivation & Objective

• To address the scarcity of ground-truth eye fixation data in saliency prediction by introducing a weakly pre-trained CNN framework.
• To explore the feasibility of using large-scale, unlabeled datasets like ImageNet for pre-training saliency models without human fixation annotations.
• To improve training efficiency and generalization by using pre-training as a regularizer for saliency prediction tasks.
• To demonstrate that domain-specific pre-training on ImageNet leads to faster convergence and better performance than random initialization on downstream saliency datasets.

Proposed method

• A shallow 5-layer CNN architecture is used, with three stages of CONV-ReLU-MAX_POOL followed by two fully-connected layers, including a maxout operation on the final layer.
• The model is first weakly pre-trained on ImageNet using a self-supervised objective that approximates human fixation patterns, without requiring ground-truth fixation maps.
• After pre-training, the network is fine-tuned on small-scale ground-truth fixation datasets (iSUN and SALICON) using a 1024-dimensional regression loss with element-wise squared error.
• The pre-training step initializes network weights to encode saliency-relevant features, reducing the need for extensive backpropagation during fine-tuning.
• The input image size is fixed at 128×128 RGB, and the output is a 32×32 saliency map generated via upsampling of the final 1024D vector.
• Receptive fields of 5×5, 3×3, and 3×3 are used in the first, second, and third convolutional stages, respectively, with 2×2 max-pooling layers.

Experimental results

Research questions

• RQ1 Can weakly pre-training on ImageNet significantly accelerate convergence during saliency prediction fine-tuning compared to random initialization?
• RQ2 Does pre-training on a large-scale, non-fixation dataset improve generalization and performance on downstream saliency prediction tasks?
• RQ3 To what extent does domain-specific pre-training on ImageNet outperform random weight initialization in terms of loss decay and final performance on benchmark datasets?
• RQ4 How does WEPSAM compare to state-of-the-art saliency models in terms of quantitative metrics and qualitative saliency map accuracy on the MIT300 dataset?
• RQ5 Can a lightweight CNN architecture achieve competitive performance when pre-trained using weak supervision from ImageNet instead of relying solely on labeled fixation data?

Key findings

• The pre-trained WEPSAM model achieves faster convergence, with training and validation loss dropping to 7.2×10⁻³ and 7.3×10⁻³ respectively after 400 epochs, compared to 7.7×10⁻³ and 7.8×10⁻³ for a randomly initialized network.
• At the start of training, WEPSAM's training and validation loss were 8.2×10⁻³ and 8.4×10⁻³, respectively, showing a 2.4×10⁻³ improvement over the randomly initialized network’s 10.6×10⁻³ and 9.6×10⁻³.
• On the MIT300 dataset, WEPSAM outperforms multiple non-learning-based models (e.g., CIW, CAS, RARE-2012) on AUC-Judd, AUC-Borji, CC, SIM, KL, and NSS metrics.
• Despite using only a single 128×128 input resolution and three convolutional layers, WEPSAM surpasses the more complex MR-CNN model on several metrics, including AUC-Judd and CC.
• Qualitative results show that WEPSAM highlights semantically important regions (e.g., faces, objects of interest), while competing models produce diffused maps emphasizing edges and gradients.
• The model’s predictions after pre-training are already coherent with ground truth fixation patterns, indicating that weak pre-training effectively captures saliency-relevant features.