[Paper Review] Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers

Zhibo Yang, Sounak Mondal+5 more|arXiv (Cornell University)|Mar 16, 2023

EEG and Brain-Computer InterfacesNeuroscience3 citations

TL;DR

This paper proposes HAT (Human Attention Transformer), a unified transformer-based model that predicts both top-down and bottom-up attention scanpaths without fixation discretization. By using a foveated retina simulation and dense heatmap prediction, HAT achieves new state-of-the-art performance across target-present, target-absent, and free-viewing tasks, improving cNSS by up to 83% on COCO-Search18.

ABSTRACT

Most models of visual attention aim at predicting either top-down or bottom-up control, as studied using different visual search and free-viewing tasks. In this paper we propose the Human Attention Transformer (HAT), a single model that predicts both forms of attention control. HAT uses a novel transformer-based architecture and a simplified foveated retina that collectively create a spatio-temporal awareness akin to the dynamic visual working memory of humans. HAT not only establishes a new state-of-the-art in predicting the scanpath of fixations made during target-present and target-absent visual search and ``taskless'' free viewing, but also makes human gaze behavior interpretable. Unlike previous methods that rely on a coarse grid of fixation cells and experience information loss due to fixation discretization, HAT features a sequential dense prediction architecture and outputs a dense heatmap for each fixation, thus avoiding discretizing fixations. HAT sets a new standard in computational attention, which emphasizes effectiveness, generality, and interpretability. HAT's demonstrated scope and applicability will likely inspire the development of new attention models that can better predict human behavior in various attention-demanding scenarios. Code is available at https://github.com/cvlab-stonybrook/HAT.

Motivation & Objective

• To unify the prediction of top-down (goal-directed) and bottom-up (salience-driven) attention scanpaths in a single model.
• To overcome the limitations of fixation discretization in prior models, which causes information loss and reduces accuracy.
• To develop a computationally efficient, dense-prediction framework suitable for high-resolution inputs.
• To create a cognitively plausible model with interpretable attention mechanisms that reflect human visual working memory dynamics.

Proposed method

• HAT employs a dual-branch CNN encoder to extract multi-scale feature maps at different eccentricities, simulating a foveated retina.
• It uses a transformer encoder with learnable spatial, temporal, and scale embeddings to dynamically integrate visual information into a working memory.
• The model predicts dense fixation heatmaps at each step, avoiding coarse discretization of fixations into grid cells.
• Foveal and peripheral tokens are used to represent central and peripheral visual information, respectively, with the foveal tokens encoding history of prior fixations.
• A self-attention mechanism aggregates information from working memory to predict the next fixation location.
• The model is trained end-to-end with per-pixel supervision, enabling high-resolution, sequential scanpath prediction.

Experimental results

Research questions

• RQ1 Can a single deep learning model effectively predict both top-down and bottom-up attention scanpaths without separate architectures for each?
• RQ2 How does avoiding fixation discretization impact the accuracy and generalization of scanpath prediction models?
• RQ3 To what extent does a foveated, transformer-based working memory improve temporal and spatial awareness in scanpath prediction?
• RQ4 How do attention weights in HAT reflect cognitive plausibility and human visual memory dynamics?
• RQ5 What components of the HAT architecture are most critical for performance, and how do they contribute to interpretability?

Key findings

• HAT achieves new state-of-the-art performance on COCO-Search18 and COCO-FreeView datasets, improving cNSS by 83% in target-present, 58% in target-absent, and 72% in free-viewing settings.
• The ablation study shows that removing foveal tokens causes the largest performance drop (cIG decreases by nearly 41%), confirming their critical role in maintaining working memory.
• Spatial and temporal embeddings contribute significantly, with cIG dropping by 21% and 14% respectively when removed, highlighting the importance of location and timing awareness.
• The model's attention maps reveal that recent fixations dominate in later stages of target-absent search, resembling free-viewing patterns, confirming cognitive consistency.
• HAT's dense-prediction approach avoids discretization loss and enables high-resolution scanpath prediction, outperforming prior methods that rely on coarse action grids.
• The model demonstrates strong generalization, achieving SOTA performance across three distinct attention control regimes: target-present, target-absent, and free-viewing.