[Paper Review] SalyPath360: Saliency and Scanpath Prediction Framework for Omnidirectional Images

Mohamed Amine Kerkouri, Marouane Tliba+2 more|arXiv (Cornell University)|Dec 31, 2021

Visual Attention and Saliency Detection33 references4 citations

TL;DR

SalyPath360 proposes a novel deep learning framework that jointly predicts saliency maps and scanpaths for omnidirectional 360° images using an encoder-decoder network with attention mechanisms and an auxiliary network for fixation point prediction via SoftArgMax. The framework fuses the saliency map and scanpath-based heatmap using an adaptive joint probability model, achieving state-of-the-art performance on the Salient360! dataset in both saliency and scanpath prediction tasks.

ABSTRACT

This paper introduces a new framework to predict visual attention of omnidirectional images. The key setup of our architecture is the simultaneous prediction of the saliency map and a corresponding scanpath for a given stimulus. The framework implements a fully encoder-decoder convolutional neural network augmented by an attention module to generate representative saliency maps. In addition, an auxiliary network is employed to generate probable viewport center fixation points through the SoftArgMax function. The latter allows to derive fixation points from feature maps. To take advantage of the scanpath prediction, an adaptive joint probability distribution model is then applied to construct the final unbiased saliency map by leveraging the encoder decoder-based saliency map and the scanpath-based saliency heatmap. The proposed framework was evaluated in terms of saliency and scanpath prediction, and the results were compared to state-of-the-art methods on Salient360! dataset. The results showed the relevance of our framework and the benefits of such architecture for further omnidirectional visual attention prediction tasks.

Motivation & Objective

• To address the challenge of predicting human visual attention in omnidirectional 360° images, which is critical for optimizing VR content delivery and enhancing Quality of Experience (QoE).
• To develop a unified framework that simultaneously predicts both saliency maps and scanpaths, capturing the dynamic nature of gaze behavior in spherical visual content.
• To improve saliency prediction by integrating scanpath-derived fixation patterns and equatorial attention bias through a learned fusion mechanism.
• To evaluate the framework's effectiveness against state-of-the-art models on the Salient360! benchmark dataset.

Proposed method

• An encoder-decoder convolutional neural network with an embedded self-attention module generates the primary saliency map from omnidirectional images in EquiRectangular Projection (ERP) format.
• An auxiliary network predicts probable viewport center fixation points using the SoftArgMax function applied to feature maps, enabling scanpath generation.
• The predicted scanpath is convolved with a Gaussian filter to produce a scanpath-based saliency heatmap.
• An adaptive joint probability distribution model fuses the primary saliency map, the scanpath-based heatmap, and an equator bias map to generate a final, unbiased saliency prediction.
• The framework leverages multi-scale features and attention mechanisms to enhance representation learning for spherical image understanding.
• The model is trained and evaluated on the Salient360! dataset using standard metrics including AUC-Judd, AUC-Borji, NSS, CC, SIM, and KLD.

Experimental results

Research questions

• RQ1 Can a unified deep learning framework effectively predict both saliency and scanpath in omnidirectional 360° images?
• RQ2 How does integrating scanpath predictions and equatorial bias improve saliency map accuracy compared to standalone saliency models?
• RQ3 To what extent does the adaptive fusion of multiple saliency representations outperform existing state-of-the-art methods on benchmark datasets?
• RQ4 What is the contribution of the SoftArgMax-based auxiliary network in generating plausible and spatially coherent scanpaths?

Key findings

• SalyPath360 achieved the highest performance on the Jarodzka and NSS metrics, outperforming state-of-the-art models such as SaltiNet and PathGan.
• The framework achieved an AUC-Judd score of 0.8610, AUC-Borji of 0.8199, and NSS of 1.8552 on the Salient360! dataset, demonstrating superior predictive accuracy.
• The one-way ANOVA test confirmed statistically significant differences (p < 0.05) between SalyPath360 and both SaltiNet and PathGan in terms of Jarodzka scores.
• The joint probability fusion module significantly improved performance, with the final map (SalyPath360) outperforming both the standalone saliency map (S) and the merged map without equator bias.
• The scanpath-based saliency map alone showed strong performance on location-based metrics but weaker results on distribution-based metrics, highlighting the need for fusion.
• Qualitative results confirmed that the predicted scanpath spans salient and equatorial regions, though the metric-based evaluation (Jarodzka) showed a disconnect from visual plausibility, suggesting limitations in that metric.