[Paper Review] SalGAN: Visual Saliency Prediction with Generative Adversarial Networks

Junting Pan, Cristian Canton-­Ferrer+5 more|arXiv (Cornell University)|Jan 4, 2017

Visual Attention and Saliency Detection23 references336 citations

TL;DR

SalGAN uses an adversarial (GAN) loss combined with a BCE content loss to predict visual saliency maps, achieving state-of-the-art results on MIT300 and SALICON across multiple metrics.

ABSTRACT

We introduce SalGAN, a deep convolutional neural network for visual saliency prediction trained with adversarial examples. The first stage of the network consists of a generator model whose weights are learned by back-propagation computed from a binary cross entropy (BCE) loss over downsampled versions of the saliency maps. The resulting prediction is processed by a discriminator network trained to solve a binary classification task between the saliency maps generated by the generative stage and the ground truth ones. Our experiments show how adversarial training allows reaching state-of-the-art performance across different metrics when combined with a widely-used loss function like BCE. Our results can be reproduced with the source code and trained models available at https://imatge-upc.github.io/saliency-salgan-2017/.

Motivation & Objective

• Motivate saliency prediction with a data-driven loss that aligns with multiple saliency metrics rather than optimizing a single metric.
• Propose an encoder-decoder generator network (SalGAN) for saliency map prediction.
• Incorporate a discriminator to distinguish predicted saliency maps from ground truth to drive adversarial training.
• Demonstrate that adversarial training improves performance across several saliency metrics.
• Show that downsampling saliency maps during training reduces computation without sacrificing accuracy.

Proposed method

• Encoder-decoder generator (SalGAN) initialized from VGG-16 up to the last two conv blocks, with a symmetric decoder producing saliency maps at input resolution.
• Discriminator network processes image+saliency pairs to distinguish real vs. generated saliency maps.
• Content loss primarily BCE over pixel-wise saliency probabilities (with sigmoid outputs per pixel).
• Adversarial loss combines BCE with a generator loss term that encourages fooling the discriminator, using L(D(I, Ŝ), 1).
• Training starts with BCE-only pretraining (≈15 epochs) before introducing adversarial training, alternating updates of generator and discriminator.
• Downsampling saliency maps (e.g., 256x192 to 64x48) during training to reduce computation while preserving performance.

Experimental results

Research questions

• RQ1 Can adversarial training improve visual saliency prediction beyond traditional pixel-wise losses?
• RQ2 What is the impact of downsampling saliency maps during training on predictive accuracy and computational efficiency?
• RQ3 How does SalGAN perform across multiple saliency evaluation metrics compared to state-of-the-art methods?
• RQ4 Does combining BCE content loss with an adversarial loss yield stable and better convergence than BCE alone?

Key findings

• Adversarial training improves multiple saliency metrics compared to BCE alone on SALICON validation and MIT300 benchmarks.
• Downsampling saliency maps by a factor of 1/4 while training does not degrade performance and can improve metrics.
• The BCE content loss provides a strong initialization and stabilizes adversarial training; the combined BCE+GAN loss yields better overall performance across most metrics.
• SalGAN achieves competitive or superior results relative to recent state-of-the-art methods on SALICON test and MIT300, across several metrics.
• Qualitative results show SalGAN predicting salient regions not captured by BCE-only models and producing smoother saliency maps.
• NSS is the only metric for which adversarial training did not consistently improve performance in all experiments.