[Paper Review] Shallow and Deep Convolutional Networks for Saliency Prediction

Junting Pan, Kevin McGuinness+3 more|arXiv (Cornell University)|Mar 2, 2016

Visual Attention and Saliency DetectionComputer Science35 references132 citations

TL;DR

The paper presents two end-to-end CNNs for saliency prediction: a shallow network trained from scratch and a deeper network using transferred bottom layers, evaluated across multiple saliency benchmarks and datasets.

ABSTRACT

The prediction of salient areas in images has been traditionally addressed with hand-crafted features based on neuroscience principles. This paper, however, addresses the problem with a completely data-driven approach by training a convolutional neural network (convnet). The learning process is formulated as a minimization of a loss function that measures the Euclidean distance of the predicted saliency map with the provided ground truth. The recent publication of large datasets of saliency prediction has provided enough data to train end-to-end architectures that are both fast and accurate. Two designs are proposed: a shallow convnet trained from scratch, and a another deeper solution whose first three layers are adapted from another network trained for classification. To the authors knowledge, these are the first end-to-end CNNs trained and tested for the purpose of saliency prediction.

Motivation & Objective

• Move from hand-crafted features to data-driven saliency prediction using CNNs.
• Develop and compare two end-to-end architectures (shallow from scratch, deep with transfer learning).
• Evaluate saliency prediction performance across multiple large-scale and benchmark datasets.
• Analyze memory requirements and training considerations for shallow vs. deep models.

Proposed method

• Shallow convnet: five learned layers (three conv, two fully connected), trained from scratch on SALICON and iSUN datasets; 64.4M parameters; output saliency map resized to input with Gaussian post-filtering.
• Deep convnet: ten weight layers with first three initialized from pre-trained VGG_M; deconvolution layer to produce saliency map; trained with Euclidean loss on SALICON data; uses transfer learning to regularize.
• Training specifics: SGD with Nesterov momentum, learning rate scheduling, 80/20 train/validation split for SALICON, downsampling to 320x240, batch size 2, 24k iterations, mean-centered inputs and targets, standard L2 weight decay.
• Evaluation: assessed on MIT Saliency Benchmark, LSUN challenge, SALICON, iSUN, MIT300 datasets; memory usage and inference considerations discussed.

Experimental results

Research questions

• RQ1 Can end-to-end CNNs predict pixel-wise saliency maps effectively compared to hand-crafted methods?
• RQ2 How do shallow-from-scratch and deep-transfer CNN architectures compare in saliency prediction performance?
• RQ3 Does transfer learning from classification nets improve saliency prediction when data is limited?
• RQ4 How well do the proposed models generalize across different saliency datasets and benchmarks?

Key findings

• Both architectures achieve competitive results on standard saliency benchmarks (e.g., iSUN validation AUC: Deep 0.63, Shallow 0.64; Judd/ Borji/ shuffled variants shown in Table 4).
• The shallow convnet won the 2015 LSUN saliency prediction challenge, outperforming several baselines on SALICON and iSUN test sets.
• On MIT300, the deep convnet attains performance comparable to DeepGaze 1, while the shallow convnet remains a strong top-performer across the benchmark set.
• Deep network benefits from transferring lower convolutional layers from a pre-trained VGG_M, improving regularization and final saliency prediction quality.
• Shallow network uses fewer layers but more parameters (due to fully connected layers) and demonstrates faster test-time processing and different spatial bias (central tendency).
• Memory analysis shows distinct trade-offs: shallow network has higher parameter count but lower blob data memory, enabling larger batch testing; deep network has lower parameter count but higher intermediate data requirements.