[Paper Review] Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual Recognition

Stefan Mathe, Cristian Sminchisescu|arXiv (Cornell University)|Dec 29, 2013

Visual Attention and Saliency Detection42 references4 citations

TL;DR

This paper introduces large-scale dynamic gaze datasets from human eye tracking during video action recognition tasks on Hollywood-2 and UCF Sports, enabling the development of learned saliency models that predict human fixations. By integrating these saliency models into end-to-end trainable computer vision systems, the authors achieve state-of-the-art recognition performance, demonstrating that human-like attention patterns can significantly improve visual action recognition accuracy.

ABSTRACT

Systems based on bag-of-words models from image features collected at maxima of sparse interest point operators have been used successfully for both computer visual object and action recognition tasks. While the sparse, interest-point based approach to recognition is not inconsistent with visual processing in biological systems that operate in `saccade and fixate' regimes, the methodology and emphasis in the human and the computer vision communities remains sharply distinct. Here, we make three contributions aiming to bridge this gap. First, we complement existing state-of-the art large scale dynamic computer vision annotated datasets like Hollywood-2 and UCF Sports with human eye movements collected under the ecological constraints of the visual action recognition task. To our knowledge these are the first large human eye tracking datasets to be collected and made publicly available for video, vision.imar.ro/eyetracking (497,107 frames, each viewed by 16 subjects), unique in terms of their (a) large scale and computer vision relevance, (b) dynamic, video stimuli, (c) task control, as opposed to free-viewing. Second, we introduce novel sequential consistency and alignment measures, which underline the remarkable stability of patterns of visual search among subjects. Third, we leverage the significant amount of collected data in order to pursue studies and build automatic, end-to-end trainable computer vision systems based on human eye movements. Our studies not only shed light on the differences between computer vision spatio-temporal interest point image sampling strategies and the human fixations, as well as their impact for visual recognition performance, but also demonstrate that human fixations can be accurately predicted, and when used in an end-to-end automatic system, leveraging some of the advanced computer vision practice, can lead to state of the art results.

Motivation & Objective

• To bridge the gap between human visual attention and computer vision by collecting large-scale, task-controlled human eye movement data during video action recognition.
• To develop novel consistency and alignment measures for analyzing spatial and sequential patterns in human fixations across subjects and videos.
• To create end-to-end trainable computer vision systems that leverage predicted human saliency for improved action recognition performance.
• To evaluate the impact of human fixation patterns on recognition accuracy and compare them with traditional computer vision sampling strategies.

Proposed method

• Collected dynamic eye-tracking data from 16 subjects viewing 497,107 frames from Hollywood-2 and UCF Sports video datasets under controlled action recognition tasks.
• Introduced sequential consistency and alignment measures to quantify spatial and temporal stability of fixation patterns across subjects.
• Trained end-to-end visual recognition pipelines using saliency-based sampling strategies derived from human fixations, including predicted and ground-truth saliency maps.
• Employed bag-of-visual-words and second-order pooling frameworks with descriptors from Harris corners, uniform sampling, and saliency-driven sampling.
• Used covariance matrices of descriptors with matrix logarithm and power scaling to enable efficient, non-linear feature encoding without additional kernels.
• Evaluated performance using leave-one-out cross-validation with 10 random seeds to assess variance and robustness of results.

Experimental results

Research questions

• RQ1 How consistent are human fixation patterns across subjects during dynamic video action recognition tasks?
• RQ2 To what extent do task constraints influence the spatial and sequential structure of human eye movements in video?
• RQ3 Can saliency models trained on human eye-tracking data accurately predict fixation locations in video stimuli?
• RQ4 How does saliency-based sampling compare to traditional interest point or uniform sampling in terms of recognition accuracy?
• RQ5 Can end-to-end trainable systems using predicted human saliency achieve state-of-the-art performance in visual action recognition?

Key findings

• Human fixation patterns exhibit high spatial and sequential consistency across subjects, even under task constraints, indicating stable visual search behavior.
• The proposed saliency models trained on human gaze data achieve high accuracy in predicting fixation locations in video stimuli.
• Using predicted saliency for sampling interest points in a bag-of-visual-words framework improved recognition accuracy to 87.5% on the UCF Sports dataset, surpassing baseline methods.
• The standard deviation of recognition accuracy across 10 random seeds was below 0.8%, indicating low variance and high robustness of the proposed pipeline.
• Saliency-based sampling outperformed traditional methods such as Harris corners (84.3%) and uniform sampling (83.9%), demonstrating the value of human attention priors.
• Second-order pooling with saliency-based sampling achieved state-of-the-art performance, confirming that human-like attention mechanisms can enhance computer vision systems.