[Paper Review] Learning a Saliency Evaluation Metric Using Crowdsourced Perceptual Judgments

Changqun Xia, Jia Li+2 more|arXiv (Cornell University)|Jun 27, 2018

Visual Attention and Saliency Detection40 references3 citations

TL;DR

This paper proposes CPJ, a deep learning-based saliency evaluation metric trained on crowdsourced perceptual judgments of saliency map similarity. Using a two-stream CNN to learn human perception factors from 134,400 binary comparisons across 16 subjects, CPJ achieves 99.6% accuracy on synthetic data, outperforming existing metrics by aligning closely with human perception of saliency map similarity.

ABSTRACT

In the area of human fixation prediction, dozens of computational saliency models are proposed to reveal certain saliency characteristics under different assumptions and definitions. As a result, saliency model benchmarking often requires several evaluation metrics to simultaneously assess saliency models from multiple perspectives. However, most computational metrics are not designed to directly measure the perceptual similarity of saliency maps so that the evaluation results may be sometimes inconsistent with the subjective impression. To address this problem, this paper first conducts extensive subjective tests to find out how the visual similarities between saliency maps are perceived by humans. Based on the crowdsourced data collected in these tests, we conclude several key factors in assessing saliency maps and quantize the performance of existing metrics. Inspired by these factors, we propose to learn a saliency evaluation metric based on a two-stream convolutional neural network using crowdsourced perceptual judgements. Specifically, the relative saliency score of each pair from the crowdsourced data is utilized to regularize the network during the training process. By capturing the key factors shared by various subjects in comparing saliency maps, the learned metric better aligns with human perception of saliency maps, making it a good complement to the existing metrics. Experimental results validate that the learned metric can be generalized to the comparisons of saliency maps from new images, new datasets, new models and synthetic data. Due to the effectiveness of the learned metric, it also can be used to facilitate the development of new models for fixation prediction.

Motivation & Objective

• To address the gap between computational evaluation metrics and human perception in saliency map comparison.
• To identify key perceptual factors influencing human judgment of saliency map similarity through extensive crowdsourced subjective testing.
• To develop a new evaluation metric that learns from human perceptual judgments rather than relying solely on statistical or information-theoretic measures.
• To create a metric that generalizes across new images, datasets, models, and synthetic data, serving as a complement to existing metrics.

Proposed method

• Conducts large-scale crowdsourced perceptual judgment tasks with 16 subjects, presenting pairs of estimated saliency maps (ESMs) and a ground-truth map (GSM) visualized via histogram equalization.
• Collects 134,400 binary annotations to determine which ESM is more similar to the GSM, forming a dataset of relative perceptual preferences.
• Designs a two-stream convolutional neural network (CPJ) that takes two ESMs and a GSM as input, learning to predict which ESM is more perceptually similar to the GSM.
• Trains CPJ using contrastive loss based on the relative perceptual judgments, encouraging the network to assign higher scores to the ESM judged more similar by humans.
• Quantifies the performance of existing metrics using the same crowdsourced data to benchmark CPJ against state-of-the-art metrics.
• Validates CPJ on unseen data, including new images, datasets, models (e.g., HFT, SP), and synthetic data, to assess generalization and robustness.

Experimental results

Research questions

• RQ1 What perceptual factors do humans consistently use when judging the similarity between saliency maps?
• RQ2 How well can a deep learning model trained on crowdsourced perceptual judgments generalize to new images, datasets, and models?
• RQ3 Can a learned metric based on human perception outperform traditional computational metrics in evaluating saliency map similarity?
• RQ4 To what extent does the learned metric CPJ align with human perception across diverse saliency map pairs, including synthetic data?

Key findings

• CPJ achieves 99.6% accuracy in correctly identifying the ground-truth saliency map as more similar than synthetic ESMs in a synthetic data test, demonstrating strong alignment with human perception.
• On synthetic data where the ground-truth is expected to outperform all others, CPJ achieves 99.6% accuracy, confirming its ability to capture the upper-bound performance correctly.
• When evaluating against a random baseline (lower bound), CPJ achieves 94.3% accuracy, showing consistent performance even when the correct choice is less obvious.
• CPJ generalizes effectively to new images, new datasets, new models (e.g., HFT, SP), and synthetic data, outperforming ten representative existing metrics.
• The top-performing existing metric only reached 72.8% accuracy in matching human judgments, highlighting the significant gap CPJ closes.
• The CNN-based framework successfully learns shared perceptual factors across 16 subjects, enabling a robust, perception-aligned evaluation metric that complements traditional metrics.