[Paper Review] Automatic Gaze Analysis: A Survey of Deep Learning based Approaches

Shreya Ghosh, Abhinav Dhall+3 more|arXiv (Cornell University)|Aug 12, 2021

Gaze Tracking and Assistive Technology199 references16 citations

TL;DR

This survey provides a comprehensive review of deep learning-based approaches for automatic gaze analysis, focusing on unsupervised, self-supervised, and weakly supervised methods. It identifies key challenges in unconstrained environments and proposes future directions for robust, real-time gaze estimation in AR/VR, HCI, and computer vision applications.

ABSTRACT

Eye gaze analysis is an important research problem in the field of Computer Vision and Human-Computer Interaction. Even with notable progress in the last 10 years, automatic gaze analysis still remains challenging due to the uniqueness of eye appearance, eye-head interplay, occlusion, image quality, and illumination conditions. There are several open questions, including what are the important cues to interpret gaze direction in an unconstrained environment without prior knowledge and how to encode them in real-time. We review the progress across a range of gaze analysis tasks and applications to elucidate these fundamental questions, identify effective methods in gaze analysis, and provide possible future directions. We analyze recent gaze estimation and segmentation methods, especially in the unsupervised and weakly supervised domain, based on their advantages and reported evaluation metrics. Our analysis shows that the development of a robust and generic gaze analysis method still needs to address real-world challenges such as unconstrained setup and learning with less supervision. We conclude by discussing future research directions for designing a real-world gaze analysis system that can propagate to other domains including Computer Vision, Augmented Reality (AR), Virtual Reality (VR), and Human Computer Interaction (HCI). Project Page: https://github.com/i-am-shreya/EyeGazeSurvey}{https://github.com/i-am-shreya/EyeGazeSurvey

Motivation & Objective

• To analyze recent advances in deep learning-based gaze estimation and segmentation, especially in weakly and unsupervised settings.
• To identify effective techniques for gaze analysis under real-world, unconstrained conditions with limited supervision.
• To evaluate current methods using domain-specific metrics and benchmark protocols tailored for gaze analysis.
• To explore future research directions for robust, low-latency gaze estimation in AR/VR and human-computer interaction.
• To bridge gaps between traditional model-based approaches and data-driven deep learning by proposing hybrid learning frameworks.

Proposed method

• Systematic review of 100+ recent works in gaze estimation, segmentation, and tracking using deep learning.
• Categorization of methods based on supervision level: fully supervised, weakly supervised, self-supervised, and unsupervised.
• Analysis of key components: eye detection (registration), feature representation (e.g., CNNs, ViTs), and gaze prediction (regression or classification).
• Evaluation of methods using standard benchmarks such as CAVE, MPII, ETH-XGaze, and TabletGaze with metrics like mean absolute error (MAE) and angular error.
• Survey of data collection devices including RGB/IR cameras, laptop/webcams, and specialized eye trackers (e.g., video-oculography).
• Integration of insights from computer vision, AR/VR, and HCI to propose hybrid models combining geometric eye models with deep appearance features.

Figure 1: A brief chronology of seminal gaze analysis works. The very first gaze pattern modelling dates back to the work of Javal et al. in 1879 [ 4 ] . One of the first deep learning driven appearance based gaze estimation models was proposed in $2015$ [ 24 ] .

Experimental results

Research questions

• RQ1 What are the most effective deep learning architectures and training paradigms for gaze estimation in unconstrained environments?
• RQ2 How do unsupervised and weakly supervised methods compare to fully supervised approaches in terms of accuracy and generalization?
• RQ3 What are the key challenges in real-world gaze estimation, and how do current methods address them?
• RQ4 How can gaze inference be leveraged to infer cognitive and affective states beyond simple gaze direction?
• RQ5 What role can multi-modal or cross-modal inputs (e.g., audio, head pose) play in improving gaze estimation robustness?

Key findings

• Unsupervised and self-supervised methods show promise in reducing reliance on costly, error-prone human-annotated gaze labels.
• Current state-of-the-art models achieve mean absolute errors below 1.5 degrees on benchmark datasets like ETH-XGaze under controlled conditions.
• Hybrid models combining geometric eye models with deep appearance features improve generalization across diverse head poses and lighting conditions.
• Future gaze trajectory prediction is emerging as a key enabler for low-latency foveated rendering in AR/VR applications.
• Multi-modal approaches using audio or head motion cues can improve gaze estimation in low-visibility or occluded scenarios.
• Despite progress, robustness in unconstrained, real-world settings—especially with extreme head poses and occlusions—remains a major open challenge.

Figure 2: Top Left: Overview of the human visual system, eye modelling and eye movement. For computer vision based automated gaze analysis, we consider an image containing eyes (left) as input. Thus, such methods analyze the visible eye regions (middle) and predict the 2-D/3-D gaze vector as output.