[Paper Review] Supervising Neural Attention Models for Video Captioning by Human Gaze Data

Youngjae Yu, Jongwook Choi+4 more|arXiv (Cornell University)|Jul 19, 2017

Multimodal Machine Learning Applications6 references3 citations

TL;DR

This paper proposes Gaze Encoding Attention Network (GEAN), a video captioning model that leverages human gaze tracking data to explicitly supervise spatial and temporal attention mechanisms. By integrating gaze-predicted visual attention maps into the encoder and decoder, GEAN achieves state-of-the-art performance in both video captioning and gaze prediction across multiple datasets, significantly improving caption quality as validated by both automatic metrics and human evaluation.

ABSTRACT

The attention mechanisms in deep neural networks are inspired by human's attention that sequentially focuses on the most relevant parts of the information over time to generate prediction output. The attention parameters in those models are implicitly trained in an end-to-end manner, yet there have been few trials to explicitly incorporate human gaze tracking to supervise the attention models. In this paper, we investigate whether attention models can benefit from explicit human gaze labels, especially for the task of video captioning. We collect a new dataset called VAS, consisting of movie clips, and corresponding multiple descriptive sentences along with human gaze tracking data. We propose a video captioning model named Gaze Encoding Attention Network (GEAN) that can leverage gaze tracking information to provide the spatial and temporal attention for sentence generation. Through evaluation of language similarity metrics and human assessment via Amazon mechanical Turk, we demonstrate that spatial attentions guided by human gaze data indeed improve the performance of multiple captioning methods. Moreover, we show that the proposed approach achieves the state-of-the-art performance for both gaze prediction and video captioning not only in our VAS dataset but also in standard datasets (e.g. LSMDC and Hollywood2).

Motivation & Objective

• To investigate whether explicit supervision using human gaze tracking data can improve attention mechanisms in video captioning.
• To address the lack of datasets that combine video captions with human gaze annotations.
• To develop a novel video captioning model that effectively fuses gaze-predicted spatial attention with temporal language modeling.
• To evaluate the impact of gaze supervision on caption quality using both automatic metrics and human annotation via Amazon Mechanical Turk.

Proposed method

• Proposes the Gaze Encoding Attention Network (GEAN), which integrates gaze-predicted attention maps into the visual feature encoding process.
• Employs a Recurrent Gaze Prediction (RGP) model to generate spatial gaze maps from video frames, which are used as supervision for attention.
• Uses a multi-stream visual encoder that processes three feature types: GoogLeNet (C3D), Scene recognition, and gaze maps from RGP.
• Applies a dual-attention mechanism: spatial attention guided by gaze maps and temporal attention in the decoder for word generation.
• Trains the model end-to-end with cross-entropy loss for captioning and L2 loss for gaze prediction, enabling joint optimization.
• Utilizes transfer learning by pretraining the RGP model on VAS and Hollywood2 datasets before fine-tuning on LSMDC.

Experimental results

Research questions

• RQ1 Can explicit supervision using human gaze data improve the performance of attention-based video captioning models?
• RQ2 Does incorporating gaze-predicted spatial attention lead to more accurate and human-preferred captions?
• RQ3 Can a model trained on limited gaze-labeled data generalize to zero-gaze datasets like LSMDC?
• RQ4 How does gaze supervision affect both gaze prediction accuracy and video captioning quality simultaneously?

Key findings

• GEAN achieves state-of-the-art performance on both video captioning and gaze prediction across the VAS, LSMDC, and Hollywood2 datasets.
• The model with RGP-based gaze supervision outperforms all baselines in both automatic metrics (BLEU, METEOR, CIDEr) and human evaluation on AMT.
• Human preference studies show a statistically significant margin in favor of GEAN-generated captions over baselines, with preference margins exceeding those indicated by automatic metrics.
• The RGP model trained on VAS and Hollywood2 generalizes well, improving captioning performance on LSMDC even without gaze annotations in the target dataset.
• The best performance is achieved when all three visual feature streams (C3D, Scene, and gaze maps) are used, confirming the complementary role of gaze supervision.
• Among fixed gaze patterns, uniform gaze weighting performs best, indicating that attending to the full visual field is more effective than biased or random attention.