[Paper Review] Following Gaze Across Views

Adrià Recasens, Carl Vondrick+2 more|arXiv (Cornell University)|Dec 9, 2016

Visual Attention and Saliency Detection24 references3 citations

TL;DR

This paper proposes an end-to-end deep learning model for following human gaze across multiple views of the same scene, leveraging saliency, gaze pose, and geometric relationships between views. Trained only on gaze annotations, the model learns to predict gaze locations in a second view even when the person is looking outside the frame, achieving state-of-the-art performance on the newly introduced VideoGaze dataset with a 0.255 mean average precision on the full test set.

ABSTRACT

Following the gaze of people inside videos is an important signal for understanding people and their actions. In this paper, we present an approach for following gaze across views by predicting where a particular person is looking throughout a scene. We collect VideoGaze, a new dataset which we use as a benchmark to both train and evaluate models. Given one view with a person in it and a second view of the scene, our model estimates a density for gaze location in the second view. A key aspect of our approach is an end-to-end model that solves the following sub-problems: saliency, gaze pose, and geometric relationships between views. Although our model is supervised only with gaze, we show that the model learns to solve these subproblems automatically without supervision. Experiments suggest that our approach follows gaze better than standard baselines and produces plausible results for everyday situations.

Motivation & Objective

• To address the challenge of following human gaze across multiple camera views in unconstrained scenes, where gaze may extend beyond the source view.
• To develop a method that predicts gaze locations in a target view using only gaze annotations, without explicit supervision for pose or geometry.
• To create a large-scale benchmark dataset for multi-view gaze following to enable training and evaluation of such models.
• To demonstrate that the model implicitly learns to solve sub-problems like saliency estimation and geometric transformation through end-to-end training.

Proposed method

• The model uses a multi-pathway architecture with separate branches for saliency prediction, gaze direction estimation, and geometric transformation between views.
• It jointly optimizes for gaze prediction using only gaze annotations, allowing the network to self-supervise on saliency and pose estimation.
• The geometric pathway estimates the transformation (rotation and translation) between camera views, enabling projection of gaze from source to target view.
• The model is trained end-to-end on the VideoGaze dataset, which contains 47,456 annotated instances of head, eyes, and gaze across video frames.
• A variant of the model is extended to detect scene changes by adding a scene-change detection head, trained on frames from different scenes.
• Performance is evaluated using mean average precision (mAP), with ablation studies to assess contributions of each pathway.

Experimental results

Research questions

• RQ1 Can a deep learning model predict gaze locations in a second view using only gaze annotations, without explicit supervision for head pose or geometry?
• RQ2 To what extent can the model implicitly learn saliency, gaze direction, and geometric relationships between views through end-to-end training?
• RQ3 How well does the model generalize to views that are temporally distant or geometrically dissimilar?
• RQ4 Can the model detect when a gaze target is outside the scene, indicating a scene change?

Key findings

• The proposed model achieves a mean average precision of 0.255 on the full test set of the VideoGaze dataset, significantly outperforming all baselines.
• On a reduced test set with frames temporally distant (1 second or more) from the source, the model achieves 0.255 mAP, while the static baseline model degrades to 0.187.
• The model performs better on views that are more different, as the saliency pathway becomes more critical in such cases.
• The ablation study shows that the best performance is achieved with only vertical-axis rotation and translation, consistent with typical camera movements.
• The extended model for scene change detection achieves a mean average precision of 0.877, well above the random chance of 0.5.
• Visualization of internal pathways confirms that the model learns to estimate geometric relationships, gaze direction, and salient regions without explicit supervision.