[Paper Review] DeepGaze II: Reading fixations from deep features trained on object recognition

Matthias Kümmerer, Thomas S. A. Wallis+1 more|arXiv (Cornell University)|Oct 5, 2016

Advanced Image and Video Retrieval TechniquesComputer Science5 references260 citations

TL;DR

DeepGaze II predicts saliency by applying a nonlinear readout on fixed VGG-19 features pretrained on SALICON, achieving state-of-the-art information gain and MIT300 AUC/sAUC performance without retraining the base network.

ABSTRACT

Here we present DeepGaze II, a model that predicts where people look in images. The model uses the features from the VGG-19 deep neural network trained to identify objects in images. Contrary to other saliency models that use deep features, here we use the VGG features for saliency prediction with no additional fine-tuning (rather, a few readout layers are trained on top of the VGG features to predict saliency). The model is therefore a strong test of transfer learning. After conservative cross-validation, DeepGaze II explains about 87% of the explainable information gain in the patterns of fixations and achieves top performance in area under the curve metrics on the MIT300 hold-out benchmark. These results corroborate the finding from DeepGaze I (which explained 56% of the explainable information gain), that deep features trained on object recognition provide a versatile feature space for performing related visual tasks. We explore the factors that contribute to this success and present several informative image examples. A web service is available to compute model predictions at http://deepgaze.bethgelab.org.

Motivation & Objective

• Demonstrate that fixed deep features from object recognition can serve as a powerful saliency space without fine-tuning.
• Quantify the information-theoretic performance (information gain explained) of the model on benchmark datasets.
• Evaluate performance on MIT300 and compare with prior saliency models.
• Show the impact of pretraining and feature type on saliency prediction.
• Model centre bias explicitly and assess its effect on predictions.

Proposed method

• Use VGG-19 features (conv5_1, relu5_1, relu5_2, conv5_3, relu5_4) mapped to a common resolution.
• Train a four-layer 1x1 convolution readout network on top of fixed VGG features to produce a saliency score O(x,y).
• Convolve O(x,y) with a Gaussian and add a centre bias prior before applying a softmax to obtain a probability map p(x,y).
• Train with maximum likelihood (log-likelihood) using a probabilistic framework and information gain as evaluation metric.
• Pretrain readout on SALICON, then fine-tune with image-wise cross-validation on MIT1003; evaluate on MIT300 held-out set.
• Use 1x1 convolutions to limit learned features to a pointwise nonlinearity on VGG features.

Experimental results

Research questions

• RQ1 Can fixed deep features from object recognition (VGG-19) provide a strong, transferable space for saliency prediction without retraining the feature extractor?
• RQ2 How much of the explainable information gain can a saliency model achieve using pretrained deep features with a learned readout?
• RQ3 What is the contribution of pretraining (SALICON) and feature choice (VGG vs AlexNet) to saliency performance?
• RQ4 How does the proposed probabilistic readout with center bias fare against benchmark saliency metrics on MIT300?
• RQ5 What qualitative insights emerge from comparing DeepGaze II predictions with gold-standard fixations?

Key findings

• DeepGaze II accounts for 87% of the explainable information gain on a MIT1003 subset, a major improvement over prior DeepGaze I (56%).
• On MIT300, DeepGaze II achieves top AUC and shuffled AUC in the MIT Saliency Benchmark (88% AUC, 77% sAUC with center bias included).
• DeepGaze II reaches near-gold-standard performance, with no images where its predictions are worse than the baseline center bias on the evaluated subset.
• Using pretrained VGG features and SALICON pretraining are the largest contributors to performance gains over DeepGaze I.
• The model maintains strong performance without retraining VGG features, relying on a small 1x1 readout and probabilistic formulation.